PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of mhealthJMIR PublicationsJMIR mHealth and uHealthAboutSearchArchiveCurrent IssueSubmitEditorial Board
 
JMIR Mhealth Uhealth. 2016 Apr-Jun; 4(2): e60.
Published online 2016 June 13. doi:  10.2196/mhealth.4936
PMCID: PMC4923590

Stepwise Development of a Text Messaging-Based Bullying Prevention Program for Middle School Students (BullyDown)

Monitoring Editor: Gunther Eysenbach
Reviewed by Christina Heminger, Rebecca Haines-Saah, and Ilaria Montagni

Abstract

Background

Bullying is a significant public health issue among middle school-aged youth. Current prevention programs have only a moderate impact. Cell phone text messaging technology (mHealth) can potentially overcome existing challenges, particularly those that are structural (e.g., limited time that teachers can devote to non-educational topics). To date, the description of the development of empirically-based mHealth-delivered bullying prevention programs are lacking in the literature.

Objective

To describe the development of BullyDown, a text messaging-based bullying prevention program for middle school students, guided by the Social-Emotional Learning model.

Methods

We implemented five activities over a 12-month period: (1) national focus groups (n=37 youth) to gather acceptability of program components; (2) development of content; (3) a national Content Advisory Team (n=9 youth) to confirm content tone; and (4) an internal team test of software functionality followed by a beta test (n=22 youth) to confirm the enrollment protocol and the feasibility and acceptability of the program.

Results

Recruitment experiences suggested that Facebook advertising was less efficient than using a recruitment firm to recruit youth nationally, and recruiting within schools for the pilot test was feasible. Feedback from the Content Advisory Team suggests a preference for 2-4 brief text messages per day. Beta test findings suggest that BullyDown is both feasible and acceptable: 100% of youth completed the follow-up survey, 86% of whom liked the program.

Conclusions

Text messaging appears to be a feasible and acceptable delivery method for bullying prevention programming delivered to middle school students.

Keywords: bullying, mhealth, text messaging, youth, prevention

Introduction

Background

Bullying is a significant adolescent health issue: depending on the type of bullying, an estimated 9% to 38% of middle school students bully their peers sometimes or more often each semester [1]. Defined as intentional peer aggression that occurs repetitively, over time, between at least two people for whom differential power exists [2], bullying is associated with negative psychosocial correlates, including externalizing problems (eg, alcohol use) for bullies and internalizing behaviors (eg, depressive symptomatology) [3-5], and suicidal ideation and attempts for victims [5,6]. Emerging research suggests that bullying also has a negative physiological impact: victims have greater changes in C-reactive protein as they age into adulthood [7]. Research also suggests dysregulation of the hypothalamic-pituitary-adrenal axis among adults who were victims of bullying, resulting in memory deficits [8]. Recent work further notes shorter telomeres, which implicates a shorter life expectancy, among those exposed to violence (eg, bullying victimization) as children compared with nonexposed youth [9].

The Current State of Bullying Prevention Programs

Existing bullying prevention programs seem to be having a modest impact. [10-16]. Existing approaches, which rely on teachers to deliver proscriptive content, may cause youth to dismiss the messages as a way of rejecting authority and exerting control over their social selves [16]. Programs that are able to remove the teacher from the content delivery may be able to overcome this challenge [17].

The Potential for Technology to Affect Behavior Change

Many are exploring the potential for digital technology to enhance the educational experience and to promote prosocial behaviors [18,19]. The wide adoption of cell phones provides novel opportunities to “go where youth are.” Indeed, 88% of 12- to 17-year-old youth have access to a cell phone. Ninety percent of these youth use text messaging (short message service, SMS) interventions [20], which is the preferred mode of communication between peers [21]. Furthermore, text messaging is cost-effective: compared with the high personnel and infrastructure costs of in-person interventions, text messaging-based programs (ie, mHealth) are scalable and cost less than US$.02 per message to send and receive. Moreover, emerging evidence supports the efficacy of these programs [22-26].

The Theory of Social-Emotional-Learning–Based Bullying Prevention Programs

Research suggests that effective behavior change programs are guided by strong theoretical models[27]. Social-Emotional- Learning (SEL)-based programs involve “the systematic development of a core set of social and emotional skills that help children more effectively handle life challenges and thrive in both their learning and their social environments” [28]. The model has emerged from influences across different movements that focused on resiliency and teaching social and emotional competencies to children and adolescents [29]. SEL is based on many well-established theories, including theories of emotional intelligence, social and emotional competence promotion, social developmental model, social information processing, and self-management [30]. It also integrates important aspects of several other behavior change models, including the health belief model, the theory of reasoned action, problem behavior theory, and social-cognitive theory [27,31].

SEL-based programs use social skills instruction to address behavior, discipline, safety, and academics in order to help youth become more self-aware, manage their emotions, build social skills (eg, empathy, perspective-taking, respect for diversity), build friendship skills, and decrease engagement in delinquent behavior [32-34]. SEL-based bullying prevention programs [34,35] target risk and protective factors that have consistently been associated with bullying and victimization in cross-sectional and longitudinal studies [36-43]. For example, empathy has been reliably found to be negatively associated with aggression and positively associated with prosocial skills [44-46]. The inverse correlation between aggression and empathy was found to be stronger in studies that focused on the emotional component of empathy than in studies in which cognitive empathy was measured [47]. Endresen and Olweus [47] conducted a study that specifically explored the association between empathy and bullying and brought attention to attitudes toward bullying, which have demonstrated positive correlations with bullying behavior [48]. Anger and hostility routinely emerge as important correlates of bullying. In several studies of bullying behavior, anger was the strongest predictor of bullying both cross-sectionally and longitudinally [48,49]. Impulsivity also plays an important role in bullying perpetration and victimization. In a prospective study of factors that predicted bullying over a 4-month period, Espelage et al [50] found a significant association between impulsivity and bullying behavior among a sample of 214 6th grade students. In addition, bullying prevention efforts have increasingly focused on the importance of bystander intervention, with significant results [37,51].

A Description of BullyDown, a mHealth Bullying Prevention Program

SEL-based programs have reported positive results in terms of reducing bullying and other disruptive behaviors in middle school [34,52,53]. As such, we used the SEL model to guide the development of BullyDown, a mHealth bullying prevention program designed for middle school students (Figure 1).

Figure 1
Social-Emotional Learning logic model.

According to the SEL model, interactivity increases participant engagement, and new behaviors need to be practiced to be integrated into one’s behavioral script. BullyDown therefore includes several features aimed at engaging youth, while also encouraging them to practice the skills discussed in the content. Principal among these is the “Text Buddy” feature, which pairs intervention participants as “buddies,” allowing them to discuss what they learn through the program with each other. Text Buddy has been used successfully in other text messaging–based programs and is associated with behavior change [54-56]. As such, we posit that the Text Buddies feature will provide an opportunity for youth to process the program information and to practice their new skills in a safe environment. To further integrate interactivity and also ensure that youth understand the program information, youth are given the opportunity to “level up” (ie, move to the next level) at the end of each week by answering a quiz question correctly.

A third interactive feature tested for acceptability is Happy Genie, which sends intervention participants on-demand messages that provide positive encouragement. This type of feature is specific to the target audience and the topic, but is based upon other behavior change programs that have also used on-demand features [54-56].

Current Paper

Similar to iterative methods used in other mHealth programming [57,58], Ybarra and colleagues [56,59,60] have described a stepwise approach to developing and testing health behavior change content delivered via text messaging. Here, we describe the procedures and experiences encountered while developing BullyDown. Findings can inform the future design of similar programs for middle school youth, as well as those aimed at reducing bullying and other aggressive behaviors.

Methods

The Chesapeake institutional review board reviewed and approved the research protocol. We sequentially implemented five activities: (1) online focus groups (FGs), (2) ongoing content development, (3) a Content Advisory Team (CAT), and (4) an internal team test, followed by (5) a beta test. We drafted content after conducting the FGs and continued refining content during the subsequent research activities. Research materials described herein are available via the Internet [61].

The development work spanned approximately 1 year: March 2014 to May 2015. For all research activities, study eligibility criteria matched those of the intended users of the intervention. Participants were enrolled in 7th or 8th grade (of exception, 6th graders took part in the FGs), owned a cell phone, were enrolled in an unlimited text messaging plan, intended to keep the same phone number for the next 6 months, had parental permission, and provided informed assent.

Focus Groups Methods

The FGs aimed to: better understand how young people experience bullying and their prior exposure to bullying prevention programs in school, to obtain the “voice” of the target

population, and to query process issues. To gather feedback from a diverse group of youth nationally, our initial recruitment strategy relied on Facebook advertising targeted to US youth aged 13- to 14-years old (Figure 2).

Figure 2
Focus group Facebook advertisements (2a).

Upon yielding a low response rate to the Facebook advertisements, we made subsequent changes to the recruitment strategy: increasing the Facebook ad budget from US$50 to US$100 per day, reducing the number of items on the Web-based screener to lower the completion time, and adding the company logo and link to the company website in the screener to increase the website legitimacy. We also contacted partner youth organizations who agreed to advertise for the research activity. When these attempts did not noticeably increase the number of eligible screeners, we contracted a research recruitment firm.

The research team developed a script of questions to guide the FG discussions. Example questions included:

When you think of the word “bullying,” what do you think of? How (if at all) is this different from what you think of when you think about “aggression?” What about being angry? Or getting into fights with friends?

Has someone asked you to think about what it might feel like to be bullied? What are things that you do that help you understand how others might be feeling? Like how it might feel to be bullied or to bully someone?

Additionally, the FGs queried process issues, including the preferred number and timing of daily text messages and feedback on possible intervention names.

Two FGs of 20 participants each were conducted. The script was written to span 3 days of questions. We posited that youths’ experiences and ability to articulate them would vary by grade level. As such, this was our stratification variable: FG1 was conducted with 6th and 7th graders, and FG2 with 7th and 8th graders. Participants logged into the bulletin board–style FG discussion on a password-protected site two to three times each day when and where it was convenient for them. Because of its asynchronous format, participants from across a wide geographic area could be included. Participants received a US$50 Amazon gift card as an incentive for complete participation during all 3 days.

Figure 3
Focus group Facebook advertisements (2b).

Content Advisory Team (CAT) Methods

We next convened a CAT to review and gather feedback about the saliency and understandability of the program content. The CAT also explored the acceptability of the three proposed program features: (1) Text Buddy, (2) Happy Genie, and (3) level-up questions.

To gather feedback, participants, or parents of participants when only the parent email address was available, received a Word document of the draft program text messages. The instructions asked youth to review and comment on each message within each week’s module (eg, Communication). Probes were included to help give direction about what they could consider when reading the proposed content (eg, Are you able to read it and understand it easily?). Instructions also emphasized that the messages were still being developed and also necessarily short given the character limit of text messages. However, during this stage, the proposed text messages were longer than 160 characters (e.g., 180 characters), as the aim was to first identify appropriate tone and content.

Next, participants took part in a 2-day online discussion to share feedback as a group. For each weekly topic, moderators asked youth to identify their favorite, least favorite, most helpful, and least helpful messages; any messages that felt too long or short; and what they learned from the module. Moderators also asked participants about the proposed program features and to suggest three of their own Happy Genie messages. Participants earned up to a US$50 amount in an Amazon gift card, US$30 for their individual review, and US$20 for their participation in the online discussion.

Beta Test Methods

Findings were integrated into the program content, including feedback about the level-up questions and Happy Genie features. For example, to combat the judgment that bullies are “bad” (and therefore, implicitly, neither you nor your friends could be bullies) that emerged during the formative work, the level-up question during week 5 was:

Good morning and welcome to Week 5! Let's get you to Level 4. True or false: Bullies are mean to others because they are bad people. Text me back your answer.

Those who responded correctly received the following reinforcing message:

Bullies are mean for lots of reasons but that doesn't mean they are bad people. Maybe they just need help learning how to share their feelings. Hello, Level 4!

Those who responded incorrectly received the following response:

Actually, bullies can be mean for lots of reasons. That doesn't mean they are bad people. They might just need some help learning how to share their feelings.

Youth who answered incorrectly were given additional chances to answer a new question correctly before they were automatically moved on to the next level:

Let's try again. For Level 4: True or false: Young people who act “differently” are asking to be bullied.

Correct responses received the following:

Right! We're all different. Kids who act or look differently are just being themselves. They want to have friends just like everyone else. Hello, Level 4!

Incorrect responses resulted in this text:

We're all different. Teens who act or look differently are just being themselves. They want to have friends just like everyone else. Onward to Level 4!

Following a successful internal team test that confirmed the program software performed as intended and that the content was readable on cell phone screens, we conducted a beta test to confirm the feasibility of the protocol and technology in a school setting. To facilitate resolution of technology problems if needed, the beta test was conducted with participants from a low-income middle school in Illinois. Research staff went to the school in early December 2014 to screen 7th and 8th grade students from four preselected classrooms. Staff issued parental permission forms to eligible students to return in 1 week. During subsequent visits, research staff picked up the signed permission forms and facilitated enrollment (ie, obtaining informed assent from the students, helping the students complete the baseline Web-based survey in the school computer room, and verifying the students’ phones for compatibility with the software program).

Participants were randomized to either the 7.5-week intervention or control group at a 2:1 ratio. This maximized the amount of information gathered about the intervention, while also gathering sufficient data about the control group experience to determine its feasibility and acceptability. To avoid school hours, participants received one program message in the morning (7:15 AM) and the remainder of messages after school hours (between 4:00 and 9:00 PM). Intervention participants received three to six messages per day for a total of 214 messages, with an additional 15 to 29 messages from the level-up feature; were randomly assigned a Text Buddy; and had access to Happy Genie. Control participants received two messages weekly, one message encouraging thoughtful behavior (eg, A message from BullyDown: “Try doing one nice thing for someone today”) and the other thanking them for their participation, for a total of 14 messages. Control group participants did not have access to the Text Buddy and Happy Genie components.

In addition to the Web-based baseline survey, all participants were asked to complete a brief survey about their program experience, conducted via text message, every other week. We also conducted a Web-based follow-up survey 1 month post-intervention: youth received a link to the survey and two subsequent reminders via text message to facilitate self-completion. Those who did not complete the follow-up survey on their own could choose to complete the survey at school with research staff.

Participants received a US$10 cash incentive for returning the parent permission form, irrespective of whether they received permission. Participants received a US$25 Amazon gift card upon completion of the 1 month post-intervention follow-up survey.

Results

Demographic characteristics of participants in the FGs and CAT, which were enrolled nationally; and the beta test, which was fielded in Illinois, are shown in Table 1. Bullying was somewhat common. For example, 23% (5/24) of beta test participants reported bullying someone via text messaging in the past 30 days.

Table 1
Demographic characteristics of BullyDown participants by development activity.

Focus Group Results

Recruitment and Enrollment

FG activities occurred between March and May 2014. Although the Facebook ads generated a high number of impressions (343,148) and unique clicks (2334) after 10 days of advertising, only 13 screeners were received. Of these, four were from eligible youth. Reasons for ineligibility included not owning a cell phone (7/13, 54%), not being enrolled in an unlimited text messaging plan (1/13, 8%), and being too young (1/13, 8%).

The recruitment firm identified 18 participants to take part in FG1 and 19 participants in FG2. Two participants from FG1 and one from FG2 assented and enrolled, but neither completed the FG nor provided a reason for their nonparticipation.

Process Issues

In total, the FG moderators posted 23 threads, each containing between one and five questions. As predicted, participant age was an important process variable: 6th grade youth had difficulty responding to questions. Homeschooled youth did as well.

For example, in response to familiarity with bullying prevention programming at school, a 12-year-old male said: “Im homeshooled so we don't have any of that.” (Note here, and throughout, that youth quotes are presented exactly as they were typed by participants.)

Participants overwhelmingly voted for the program name “BullyDown.” As one participant shared: “I like BullyDown.com the best. It states the purpose of the website and it's easy to remember!” (13-year-old female). Youth also expressed a preference for receiving two to four program text messages per day and felt that more would be too many.

Program Content

While some youth reported that their schools had not discussed bullying, most youth were able to articulate and also agreed with school definitions of bullying: “Our school considers any type of picking on or hurting someone's feelings bullying and I feel the same.” (12-year-old male). Overall, youth related to the definition of bullying as verbal, physical, and relational forms of aggression that can happen among friends and nonfriends:

Bullying comes in many forms. Physical, mental and cyber are all kinds of bullying. To me bullying is picking on or hurting someone you know can't defend themselves. A person bullies others to feel better or more important about themselves. Being aggressive towards another person is not the same as bullying. To me aggression is more about anger and the anger one feels towards another.

13-year-old female

Youth perceptions of victims varied, and in some cases, youth felt that some victims behaved in ways that contributed to their victimization. Youth also recognized that some students who are socially popular engage in bullying, making it more difficult to stand up to these bullies. Because emotion regulation is a core component of bullying prevention, we also asked FG participants about coping strategies they used when stressed or upset. Answers included: reading/writing, listening to music, counting, belly breathing, and talking to siblings and parents.

Integrating Focus Group Feedback into BullyDown

Based upon the SEL model, we drafted seven modules: (1) communication, (2) coping with stress and problem solving, (3) empathy, perspective-taking, and respect for diversity, (4) recognizing bullying, attitudes toward bullying, and attitudes supportive of aggression, (5) emotion regulation: anger, hostility, and impulsivity, (6) bystanders and intentions to intervene to help others, and (7) resources and support.

Table 2
Example BullyDown program messages by SEL component.

We incorporated quotes, as well as personal experiences shared by youth, from the FGs into program messages. For example, youth responses included:

Bullying is something done to someone weaker then you that you know cant defend themselves. When you go behind the other persons back it usally means your scared to say it to there face.

14-year-old male

I saw a boy making fun of a handicapped student right in front of a teacher and they did not say anything, so I did and then they stopped it and the boy who was bullying was sent to the office.

13-year-old female

These became adapted into the respective program messages:

Bullying can even be spreading a rumor about someone (true or not). It can happen face-to-face or behind your back. It can be online or through text messaging.

Another teen said: A boy was making fun of a handicapped teen in front of a teacher, but she didn't say anything. So I did. The boy was sent to the office.

In response to a question about how bullying might make a person feel, a participant shared:

it makes you feel badly about yourself and that feeling lasts for a long time. It doesn't go away quickly, even if others try to tell you it's not true. Deep inside you feel like it is true what they are saying. It stays with you and keeps coming back and being in your mind.

12-year-old female

This was translated into the following program message:

A girl who was bullied told me: It makes you feel badly about yourself and that feeling lasts for a long time. It doesn't go away quickly……even if others try to tell you it is not true. Deep inside you feel like what they’re saying is true. It stays with you and keeps coming back into your mind.

Content Advisory Team Results

Recruitment and Retention

We conducted the CAT between August and September 2014. Given the success using a recruitment firm during the FGs, we determined this strategy to be the most efficient for subsequent research activities. We excluded 6th graders and homeschooled youth in these subsequent activities because of their difficulty responding to questions during the FGs. The recruitment firm recruited 12 youth for the CAT (Table 1). Fifty percent (6/12) of participants completed both the individual text message review and the online discussion. Twenty-five percent (3/12) of participants completed only the individual review. Of these three nonparticipants, one parent withdrew their child from the study due to illness and lack of time to complete the CAT. Another parent requested a Web-based version of the survey, which was provided although feedback was not received. The reason for the third youth’s nonparticipation is unknown.

Engagement with the CAT was low in part because of participants’ confusion about the protocol. Both the recruitment firm and the research staff received multiple requests from parents for clarification about the process. This confusion seemed to persist even after research staff explained the process to parents over the phone. Following up with participants and/or their parents was also difficult because messages from research staff often went unanswered. In many cases, the recruitment firm needed to contact parents and/or participants on behalf of research staff to resolve the issue. Response rates may also have been low because the CAT field period overlapped with the start of a new school year. After yielding little feedback, we extended the 7-day deadline by a week. To help invigorate completion, research staff provided participants with examples of the type of feedback that would be informative to program development. Interestingly, two participants requested that the messages be made available via the Internet through Google Docs. One did not have access to Word, and the other requested it for anticipated convenience.

Atypical of our past experience with adolescents [62], parents of CAT participants were often the point-of-contact. This is perhaps because the recruitment firm identified youth through their parents. As a result, we often lacked contact information for the youth. To avoid this issue in the future, researchers should consider including a study eligibility criterion that requires youth contact information or partnering with a recruitment firm that allows the project staff to talk directly to both parent and child at the outset of the study. Researchers should also be mindful of major transition periods for the target audience (eg, the beginning of a school year) when identifying desired field dates.

Text Message Review

As shown in Table 3, youth feedback to each of the messages were generally positive (eg, “Great job describing it”) and, in some cases, elicited personalized responses (eg, “I hate being accused of being bully”). Although some program messages received more negative feedback than others (eg, the message for recognizing bullying; emotion regulation: impulsivity; a quote from cartoon character BMO), at least one-half of the participants positively endorsed most messages. A participant also made a comment in the “emotion regulation: anger, hostility” section suggesting that some of the daily transitions between topics needed improvement.

Table 3
Example per-message feedback from the Content Advisory Team.

Online Discussion

During the 2-day online group discussion, participants generally rated the program modules positively: between 80% (4/5) and 100% (5/5 or 4/4, depending on the number of youth who responded to the poll) of youth liked/strongly liked them. As an exception, 75% (3/4) of youth disliked the problem-solving module. Only one youth provided details on his negative reaction to the module:

I liked the how to deal with stress messages, the tips on what to do. The examples about starting rumors was nothing I would ever do, so did not like that.

12-year-old male

When asked about their overall impression of the program, many participants responded positively:

the program is perfect no changes needed.

13-year-old male

Less enthusiastic feedback was also received:

some of the messages were too long and boring and some of them were kinda confusing

12-year-old female

it was jumping from topic to topic.

13-year-old female

Four youth had negative reactions to the tone of some messages. For example, one youth shared:

[the messages] sounded like an adult trying to be ‘cool’ - the messages were too long and wordy and sounded like things my mom would say.

12-year-old male

When asked for further detail, youth specifically disliked the use of the phrase “rock star” in one of the messages. Although many participants liked the inclusion of quotes from other teens, one youth questioned the authenticity of the quotes (ie, emotion regulation: anger, hostility message; Table 3).

The level-up feature was well received, although it too was not immune to criticism. For example, some thought that while the feature was acceptable, the questions needed to be more challenging:

the questions to get to the next level were way too easy. the second try was even more easy. you should probably make it a little harder.

12-year-old male

Another participant thought that the questions felt like homework:

… I don't like having to 'earn' levels - it makes it more like a school assignment, not something that should be have to be done. why do you have to pass a level to move on? is there a prize or something to get through the levels. If I was doing this in real life and did not pass a level, I would just stop and not continue anymore.

12-year-old male

Regarding program features, participants were enthusiastic about Happy Genie:

This genie feature looks amazing this is just what is needed when their life feels like a bore and nothing is great about it. The feature that it sends them a positive quote is very nice indeed. Overall great feature really perked up my spirits with those quotes i am sure it will do just the same for the participants.

13-year-old male

This sentiment was echoed by a 12-year-old female:

I think this feature is great! It could be really helpful for a person whose having a bad day.

Youth also suggested several name alternatives, including Forever Friend, Good Day Genie, and the Happy Doctor.

Integrating Content Advisory Team Feedback Into BullyDown

Overlapping feedback provided by two or more youth were integrated into the program content where possible. For example, we deleted the word “rock star” and messages deemed particularly confusing (eg, the quote from BMO). We also modified the tone of messages to sound less motherly or adult-like. The message:

You can tell your friend how you feel. It might be hard. And it might not go well. But if you’re in a healthy friendship and you use assertive communication, it could go great!

Subsequently became:

Tell your friend how you feel. It might be hard, and it might not go well. But if you’re in a healthy friendship, and you talk assertively, it might go great!

We also reviewed the messages scheduled for the beginning and end of each program day for their flow and transitions, based on youth feedback.

We chose not to make the level-up questions more challenging as this seemed likely to make the feature further simulate homework. The level-up protocol was already designed to automatically move nonresponsive participants to the next module if they did not respond to the questions after a certain period. We established a plan to closely monitor participant reactions to the level-up feature during the beta test for further indication of acceptability.

We deemed participants’ positive review of Happy Genie as supportive of its acceptability. We renamed the feature “Forever Friend” based on the names suggested by participants and added inspirational messages (eg, A new day = a new beginning) and quotes from famous people (eg, “I’ve failed over and over again in my life. And that is why I succeed.” – Michael Jordan).

Beta Test Results

Recruitment and Enrollment

Enrollment spanned 3 months and required research staff to visit the partnered middle school four times. Research staff screened 78 youth, 45% (35/78) of whom were eligible. Reasons for ineligibility included: not having a cell phone (21/78, 27%), not or uncertain if they were enrolled in an unlimited text messaging plan (9/78, 12%), and not planning to or uncertain if they would have the same phone number for at least 6 months (21/78, 27%), including the possibility that parents may take away phone privileges as a form of punishment.

Thirty-three eligible youth returned their permission forms, all of whom had parental permission to participate. Twenty-four eligible students provided written assent to participate, of which 22 completed the registration process and were randomized. Of the nine youth who had parental permission but did not enroll, three changed their mind, two moved, one was absent every time the research staff went to the school, and three had problems with phone access (eg, one shared the phone with a parent). Reasons for not completing the registration process included no longer being interested (n=1) and no longer having a cell phone (n=1). Participants were successfully randomized, resulting in 14 students assigned to BullyDown and eight to the control group (Table 1).

Process Issues

We conducted the beta test between December 2014 and May 2015. No concerns from the school administration or staff were expressed during the field period. Participants encountered some technology difficulties, however. The school’s firewall initially blocked the project enrollment website, but school technology staff quickly resolved the issue. Additionally, many students forgot to bring their phones to verify its compatibility with the program software. These students completed the Web-based baseline survey and were instructed to complete enrollment at home by responding with the word “verify” to the text message verification. Those randomized to the intervention group received a link to the Text Buddy Code of Conduct to read and accept. Some students had trouble completing the tasks, requiring research staff to return to the school to assist them in person.

Thirty-six percent (8/22) of participants completed the Web-based follow-up survey on their own time, and 14 completed it at school. The primary reason for not completing the survey independently was forgetting to do it. One participant thought the survey link did not look legitimate and so was hesitant to click on the link.

Program Acceptability and Feasibility

During the program delivery period, 86% (12/14) of intervention participants sent at least one message to their Text Buddy (range, 2-52), and 29% (4/14) sent at least one message to Happy Genie (range, 1-8). Half of intervention participants (7/14) responded to all seven weekly level-up questions, and 21% (3/14) responded to two or fewer level-up questions.

Feedback from the biweekly brief survey (Table 4) suggested that many intervention participants thought the messages were interesting and fun, although some felt the messages were boring (eg, week 2 messages). Seventy-five percent (9/12) of intervention youth were able to provide an example of a message they found memorable (week 4), and 75% (9/12) of youth provided a specific example of something they liked about the program (week 6).

Table 4
Beta test intervention group participant biweekly qualitative feedback.

Moreover, all participants said they would recommend the program to a friend when asked at the end of week 6 (Table 5). Opportunities for improvement were also noted: although 80% (8/10) of intervention participants agreed that reading the messages the same day they were received was easy (Table 5), five youth shared concerns about the message quantity or timing when asked at the end of week 6 about something they may not like about the program (Table 4).

Table 5
Acceptability and feasibility data from pilot test participants.

Feedback from intervention participants suggested that they were enthusiastic about having a Text Buddy. This feature also drew at least one negative response each week, however (Table 4). Subsequent follow-up with youth revealed that their discomfort centered on being paired with someone from the same school and having to talk about bullying with a schoolmate: “…A Lot of kids in this area kno each other and don't want to talk about things like this.” (12-year-old male). Some youth also expressed frustration that their buddies were unresponsive to their messages; or felt discomfort about being paired with someone of the other sex or in a different grade.

As shown in Table 5, both the intervention and control group content appeared to be acceptable: more than four in five youth in each arm agreed that they liked the program. With 88% (7/8) of control participants saying that the program helped them not bully others in the future, this arm, although of lesser intensity than the intervention arm, appeared to blind participants to their arm assignment. In addition to high scores for understandability and salience of program content, the experience also seemed to be feasible: most youth disagreed or strongly disagreed that the program sent too many messages, and none of the beta test participants in either arm agreed that they stopped reading the messages by the end of the program.

The enrollment process seemed to require in-person facilitation by research staff at a level that is at least commensurate with an in-person intervention. Both interesting and potentially problematic, 33% (11/33) of youth who were eligible and had parental permission lost interest in participating before enrollment. Given the small sample size, this may be an anomaly. Further investigation of this feasibility issue in a larger trial is needed before conclusions can be drawn. Also, some participants shared that they would have been more likely to respond to the follow-up survey if the link had been sent by email instead of text message. Future efforts should consider collecting multiple forms of contact (eg, Facebook, email address).

Despite CAT participants’ expressed interest in the concept of Text Buddy, most beta test participants only sent one or two messages to their buddy. Based upon their feedback, future trials could ideally match Text Buddies who are attending different schools and, when possible, of the same sex and grade level. An interactive guide, either in-person or via the Internet, instructing how to use the feature could also be incorporated into the content. Additionally, only participants with expressed interest in the Text Buddy feature could be matched to reduce the likelihood of unresponsive buddies.

Several intervention youth expressed concern about the intensity and/or timing of the daily messages during the field period but not when it was assessed at 1-month follow-up. This feedback most likely arose on days that had five or six messages, resulting in participants receiving four or five messages across a 4-hour window after school. To ameliorate this, the total number of messages per day in future iterations of the program will be limited to four where possible.

Consistent with feedback from some CAT participants who thought the level-up questions were too easy, one beta test participant thought that the questions should be more meaningful. Given the lack of negative feedback from other beta test participants about the level-up experience and the desire to avoid simulating homework however, more feedback from randomized controlled trial participants will be solicited before making significant changes.

Discussion

Principle Findings

Stepwise development of BullyDown over a 1-year period helped ensure the program was generally well received by middle school students. Findings reveal important insights about developing mHealth interventions for younger youth, particularly those focused on bullying prevention, including the preferred number of messages per day (ie, 2-4 messages), a desire for shorter messages, and the ideal age range (ie, 7th grade and older). In interventions aimed at older adolescents (e.g. 14- to 17-years old), others have noted a similar preference among participants for a low-intensity experience in interventions [63,64]; however, higher intensity programs (eg, 8-15 messages per day) have also received indications of acceptability [56]. The desired threshold seems likely to be affected not only by the age of the youth but also the topic in question. This variability in participants’ acceptability of various program intensities highlights the importance of iteratively developing new mHealth program using ongoing feedback from the target population.

Also contrasting other mHealth adolescent recruitment efforts [65], Facebook was less efficient in nationally recruiting youth than was a more traditional recruitment method (ie, using a recruitment firm). Initial interest in the ads was commensurate with expectations for a successful campaign, but completion of the Web-based screener seemed to be an issue. Perhaps younger youth are more cautious when providing contact information to unknown people or organizations via the Internet, or perhaps the topic (ie, bullying prevention), when more fully explained, is of little interest to youth. However, the school administrator’s enthusiasm for the beta test suggests that the program is acceptable and feasible for implementation in school-based samples. It also supports a hypothesis that the eventual program will need to be promoted to school administrators rather than youth directly.

Several additional differences between the experiences in the BullyDown beta test with the implementation of other text messaging–based health behavior change programs should be noted. For example, compared with high school-aged youth [56], middle school students' relationship with their phones seems to be less predictable: (1) they do not carry their phones everywhere – at least not to school and not necessarily outside of school, (2) they might not remember their phone number accurately, and (3) their access appears to be more tenuous – parents are more likely to restrict their cell phone use. Also, staff resources needed to enroll youth appear to be commensurate with in-person interventions. This means that cost savings in enrollment will not be realized. Once the intervention program messages begin however, most 7th and 8th graders are able to engage with the program on their own, suggesting that once engaged, a standalone technology intervention has promise. These differences highlight the heterogeneity of mHealth research: simply because a particular protocol works for one program and population does not mean the same protocol will work with other programs and populations.

Although control and intervention participants attended classes side by side, the control arm appears to have been blinded: they were equally likely as the intervention arm to say they liked the program, and a similar percent of youth said they learned valuable things about bullying during the program. This suggests that randomization at the individual-level may not pose a serious threat to contamination; however, results should be replicated in a larger sample size.

Limitations

Specific program recommendations that emerged from this work may not generalize to other health behavior change topics or youth populations. Although the national recruitment strategy is a strength, FG and CAT participants may not be representative of all youth given they were identified through a recruitment firm. Beta test participants were recruited from a disadvantaged public middle school in Illinois and may not be representative of students in other settings. More generally, youth who have a cell phone and are enrolled in an unlimited text messaging plan may be different from youth who do not meet these criteria. It seems unlikely however that youth who lack a cell phone or are not on an unlimited text messaging plan would opt into such a program if it was publicly available. Additionally, confirming that youth actually read the text messages is not possible beyond their self-report; therefore, actual exposure is unknown.

Conclusion

Increasingly, SEL programs are being implemented in schools across the United States to address a wide-range of problematic behaviors (eg, bullying, delinquency) and to promote academic success. However, the majority of these programs include curricula that range from 15 to 20 or more lessons, requiring significant instructional time and resources that some school districts may not be able to provide. BullyDown harnesses the benefits of text messaging, such as being able to go where youth are and communicating important information in consumable chunks, to offer a bullying prevention program that can be delivered outside of school and without the need for facilitator time or resources. While expensive in time and money, iterative intervention refinement increases the likelihood that the resulting intervention is salient to the target population while retaining its adherence to theory.

Given the positive feedback from beta test participants about the program content and the experience as well as the 100% (22/22) retention rate, the BullyDown mHealth bullying prevention program appears to be both feasible and acceptable for implementation in a middle school setting with 7th and 8th grade youth. Our next step will be to test the intervention in a large randomized controlled trial to see if exposure is associated with reduction in bullying behavior.

Acknowledgments

Drs. Ybarra and Espelage conceived the study. Dr. Ybarra identified the iterative development steps. Dr. Espelage mapped the SEL model to content areas and Dr. Ybarra led the writing of the content, including the integration of findings into the messaging. Drs. Ybarra and Espelage co-moderated the focus group discussions. Dr. Espelage and Ms. Prescott co-moderated the CAT. Ms. Prescott managed participant recruitment and retention activities. All three authors helped finalize the program content and contributed to the fielding of the beta test. The manuscript was drafted by Dr. Ybarra and Ms. Prescott, and significantly edited by Dr. Espelage. All authors approved the final version of the manuscript.

The project described was supported in part by a grant from HopeLab. The content is solely the responsibility of the authors and does not necessarily represent the official views of the HopeLab. We would like to thank the entire study team at the Center for Innovative Public Health Research and the University of Illinois at Urbana-Champaign for their contributions to the research and the participants for their time in the study. We also thank Ms. Emilie Chen for her assistance proofreading the article.

Abbreviations

CAT
content advisory team
FG
focus group
SEL
social-emotional-learning
SMS
short message service

Footnotes

Conflicts of Interest: None declared.

References

1. Wang J, Nansel TR, Iannotti RJ. Cyber and traditional bullying: differential association with depression. J Adolesc Health. 2011;48:415–417. doi: 10.1016/j.jadohealth.2010.07.012. http://europepmc.org/abstract/MED/21402273. [PMC free article] [PubMed] [Cross Ref]
2. Gladden R, Vivolo-Kantor A, Hamburger M, Lumpkin C. Bullying Surveillance Among Youths: Uniform Definitions for Public Health and Recommended Data Elements, Version 1. Atlanta: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention and U.S. Department of Education; 2013.
3. Sourander A, Helstelä L, Helenius H, Piha J. Persistence of bullying from childhood to adolescence--a longitudinal 8-year follow-up study. Child Abuse Negl. 2000;24:873–881. [PubMed]
4. Bond L, Carlin JB, Thomas L, Rubin K, Patton G. Does bullying cause emotional problems? A prospective study of young teenagers. BMJ. 2001;323:480–484. http://europepmc.org/abstract/MED/11532838. [PMC free article] [PubMed]
5. Arseneault L, Bowes L, Shakoor S. Bullying victimization in youths and mental health problems: 'much ado about nothing'? Psychol Med. 2010;40:717–729. doi: 10.1017/S0033291709991383. [PubMed] [Cross Ref]
6. Klomek AB, Sourander A, Gould M. The association of suicide and bullying in childhood to young adulthood: a review of cross-sectional and longitudinal research findings. Can J Psychiatry. 2010;55:282–288. [PubMed]
7. Copeland WE, Wolke D, Lereya ST, Shanahan L, Worthman C, Costello EJ. Childhood bullying involvement predicts low-grade systemic inflammation into adulthood. Proc Natl Acad Sci U S A. 2014 May 27;111:7570–7575. doi: 10.1073/pnas.1323641111. http://www.pnas.org/cgi/pmidlookup?view=long&pmid=24821813. [PubMed] [Cross Ref]
8. Vaillancourt T, Duku E, Becker S, Schmidt LA, Nicol J, Muir C, Macmillan H. Peer victimization, depressive symptoms, and high salivary cortisol predict poorer memory in children. Brain Cogn. 2011;77:191–199. doi: 10.1016/j.bandc.2011.06.012. [PubMed] [Cross Ref]
9. Shalev I, Moffitt TE, Sugden K, Williams B, Houts RM, Danese A, Mill J, Arseneault L, Caspi A. Exposure to violence during childhood is associated with telomere erosion from 5 to 10 years of age: a longitudinal study. Mol Psychiatry. 2013;18:576–581. doi: 10.1038/mp.2012.32. http://europepmc.org/abstract/MED/22525489. [PMC free article] [PubMed] [Cross Ref]
10. Merrell KW, Gueldner BA, Ross SW, Isava DM. How effective are school bullying intervention programs? A meta-analysis of intervention research. School Psychology Quarterly. 2008;23:26–42. doi: 10.1037/1045-3830.23.1.26. [Cross Ref]
11. Ttofi MM, Farrington DP. Effectiveness of school-based programs to reduce bullying: a systematic and meta-analytic review. J Exp Criminol. 2011 Sep 16;7:27–56. doi: 10.1007/s11292-010-9109-1. [Cross Ref]
12. Ferguson CJ, San Miguel C, Kilburn JC, Sanchez P. The effectiveness of school-based anti-bullying programs: a meta-analytic review. Criminal Justice Review. 2007;32:401–414. doi: 10.1177/0734016807311712. [Cross Ref]
13. Smith J, Schneider B, Smith P, Ananiadou K. The effectiveness of whole-school antibullying programs: A synthesis of evaluation research. School Psychology Review. 2004;33:547–560.
14. Vreeman RC, Carroll AE. A systematic review of school-based interventions to prevent bullying. Arch Pediatr Adolesc Med. 2007;161:78–88. doi: 10.1001/archpedi.161.1.78. [PubMed] [Cross Ref]
15. Ttofi MM, Farrington DP. Bullying prevention programs: the importance of peer intervention, disciplinary methods and age variations. J Exp Criminol. 2012;8:443–462. doi: 10.1007/s11292-012-9161-0. [Cross Ref]
16. Yeager DS, Fong CJ, Lee HY, Espelage DL. Declines in efficacy of anti-bullying programs among older adolescents: Theory and a three-level meta-analysis. Journal of Applied Developmental Psychology. 2015;37:36–51. doi: 10.1016/j.appdev.2014.11.005. [Cross Ref]
17. Paluck EL, Shepherd H. The salience of social referents: a field experiment on collective norms and harassment behavior in a school social network. J Pers Soc Psychol. 2012;103:899–915. doi: 10.1037/a0030015. [PubMed] [Cross Ref]
18. O'Bannon BW, Thomas KM. Mobile phones in the classroom: preservice teachers answer the call. Computers & Education. 2015;85:110–122. doi: 10.1016/j.compedu.2015.02.010. [Cross Ref]
19. Whittaker R, Merry S, Stasiak K, McDowell H, Doherty I, Shepherd M, Dorey E, Parag V, Ameratunga S, Rodgers A. MEMO--a mobile phone depression prevention intervention for adolescents: development process and postprogram findings on acceptability from a randomized controlled trial. J Med Internet Res. 2012;14:e13. doi: 10.2196/jmir.1857. http://www.jmir.org/2012/1/e13/ [PMC free article] [PubMed] [Cross Ref]
20. Lenhart A. Teens, Social Media, & Technology Overview. Washington, DC: Pew Internet & American Life Project; 2015. [2016-04-22]. http://www.pewinternet.org/2015/04/09/teens-social-media-technology-2015/ webcite.
21. Lenhart A, Ling R, Campbell S, Purcell K. Teens and Mobile Phones. Washington, DC: Pew Internet & American Life Project; 2010. [2016-04-22]. http://www.pewinternet.org/2010/04/20/teens-and-mobile-phones/ webcite.
22. Whittaker R, McRobbie H, Bullen C, Borland R, Rodgers A, Gu Y. Mobile phone-based interventions for smoking cessation. Cochrane Database Syst Rev. 2012;11:CD006611. doi: 10.1002/14651858.CD006611.pub3. [PubMed] [Cross Ref]
23. Preston KE, Walhart TA, O'Sullivan AL. Prompting healthy behavior via text messaging in adolescents and young adults. American Journal of Lifestyle Medicine. 2011;5:247–252. doi: 10.1177/1559827610392325. [Cross Ref]
24. Wei J, Hollin I, Kachnowski S. A review of the use of mobile phone text messaging in clinical and healthy behaviour interventions. J Telemed Telecare. 2011;17:41–48. doi: 10.1258/jtt.2010.100322. [PubMed] [Cross Ref]
25. Cole-Lewis H, Kershaw T. Text messaging as a tool for behavior change in disease prevention and management. Epidemiol Rev. 2010;32:56–69. doi: 10.1093/epirev/mxq004. http://europepmc.org/abstract/MED/20354039. [PMC free article] [PubMed] [Cross Ref]
26. Haug S, Schaub MP, Venzin V, Meyer C, John U. Efficacy of a text message-based smoking cessation intervention for young people: a cluster randomized controlled trial. J Med Internet Res. 2013;15:e171. doi: 10.2196/jmir.2636. http://www.jmir.org/2013/8/e171/ [PMC free article] [PubMed] [Cross Ref]
27. Greenberg MT, Weissberg RP, O'Brien MU, Zins JE, Fredericks L, Resnik H, Elias MJ. Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. Am Psychol. 2003;58:466–474. [PubMed]
28. Ragozzino K, Utne O’Brien M. Social and Emotional Learning and Bullying Prevention. Chicago: Collaborative for Academic, Social, and Emotional Learning (CASEL); 2009. [2016-04-22]. https://static1.squarespace.com/static/513f79f9e4b05ce7b70e9673/t/5367958ee4b0dbc1364dbb7b/1399297422536/3_SEL_and_Bullying_Prevention_2009.pdf webcite.
29. Elias M, Zins JE, Weissberg RP, Frey KS, Greenberg MT, Haynes NM, Shriver TP. Promoting social and emotional learning: guidelines for educators. Alexandria, Va: Association for Supervision and Curriculum Development; 1997.
30. Guerra NG, Bradshaw CP. Linking the prevention of problem behaviors and positive youth development: core competencies for positive youth development and risk prevention. New Dir Child Adolesc Dev. 2008;2008(122):1–17. doi: 10.1002/cd.225. [PubMed] [Cross Ref]
31. Hawkins J, Smith B, Catalano R. Social development and social and emotional learning. In: Zins JE, Weissberg RP, Wang MC, Walberg HJ, editors. Building Academic Success on Social and Emotional Learning: What Does the Research Say? New York, NY: Teachers College Press; 2004. pp. 135–150.
32. Zins JE, Weissberg RP, Wang MC, Walberg HJ. Building Academic Success on Social and Emotional Learning: What Does the Research Say? New York: Teachers College Press; 2004.
33. Brown EC, Low S, Smith BH, Haggerty KP. Outcomes from a school-randomized controlled trial of Steps to Respect: a school bullying prevention program. School Psychology Review. 2011;40:423–243.
34. Espelage DL, Low S, Polanin JR, Brown EC. The impact of a middle school program to reduce aggression, victimization, and sexual violence. J Adolesc Health. 2013;53:180–186. doi: 10.1016/j.jadohealth.2013.02.021. [PubMed] [Cross Ref]
35. Botvin GJ, Griffin KW, Nichols TD. Preventing youth violence and delinquency through a universal school-based prevention approach. Prev Sci. 2006;7:403–408. doi: 10.1007/s11121-006-0057-y. [PubMed] [Cross Ref]
36. Basile KC, Espelage DL, Rivers I, McMahon PM, Simon TR. The theoretical and empirical links between bullying behavior and male sexual violence perpetration. Aggression and Violent Behavior. 2009;14:336–347. doi: 10.1016/j.avb.2009.06.001. [Cross Ref]
37. Polanin J, Espelage D, Pigott T. A meta-analysis of school-based bullying prevention programs’ effects on bystander intervention behavior. School Psychology Review. 2012;41:47–65.
38. Hong JS, Espelage DL. A review of research on bullying and peer victimization in school: An ecological system analysis. Aggression and Violent Behavior. 2012;17:311–322. doi: 10.1016/j.avb.2012.03.003. [Cross Ref]
39. Espelage DL, Bosworth K, Simon TR. Examining the social context of bullying behaviors in early adolescence. Journal of Counseling and Development. 2000;78:326–333. doi: 10.1002/j.1556-6676.2000.tb01914.x. [Cross Ref]
40. Espelage D, Green H, Polanin J. Willingness to intervene in bullying episodes among middle school students: individual and peer-group influences. The Journal of Early Adolescence. 2011;32:776–801. doi: 10.1177/0272431611423017. [Cross Ref]
41. Holt MK, Espelage DL. Perceived social support among bullies, victims, and bully-victims. J Youth Adolescence. 2006;36:984–994. doi: 10.1007/s10964-006-9153-3. [Cross Ref]
42. Cook CR, Williams KR, Guerra NG, Kim TE, Sadek S. Predictors of bullying and victimization in childhood and adolescence: a meta-analytic investigation. School Psychology Quarterly. 2010;25:65–83. doi: 10.1037/a0020149. [Cross Ref]
43. Espelage DL, Holt MK, Henkel RR. Examination of peer-group contextual effects on aggression during early adolescence. Child Dev. 2003;74:205–220. [PubMed]
44. Feshbach N, Feshbach S. Empathy training and the regulation of aggression: potentialities and limitations. Academic Psychology Bulletin. 1982;4:399–413.
45. Kaukiainen A, Björkqvist K, Lagerspetz K, Österman K, Salmivalli C, Rothberg S, Ahlbom A. The relationships between social intelligence, empathy, and three types of aggression. Aggressive Behavior. 1999;25:81–89.
46. Miller PA, Eisenberg N. The relation of empathy to aggressive and externalizing/antisocial behavior. Psychol Bull. 1988;103:324–344. [PubMed]
47. Endresen I, Olweus D. Self-reported empathy in Norwegian adolescents: Sex differences, age trends,relationship to bullying. In: Bohart AC, Stipek DJ, editors. Constructive & Destructive Behavior: Implications for Family, School, & Society. Washington, DC: American Psychological Association; 2001. pp. 147–165.
48. Boulton MJ, Trueman M, Flemington I. Associations between secondary school pupils' definitions of bullying, attitudes towards bullying, and tendencies to engage in bullying: age and sex differences. Educational Studies. 2002;28:353–370. doi: 10.1080/0305569022000042390. [Cross Ref]
49. Bosworth K, Espelage D, Simon T. Factors associated with bullying behavior in middle school students. Journal of Early Adolescence. 1999;19:341–362.
50. Espelage DL, Bosworth K, Simon TR. Short-term stability and prospective correlates of bullying in middle-school students: an examination of potential demographic, psychosocial, and environmental influences. Violence Vict. 2001;16:411–426. [PubMed]
51. Salmivalli C. Bullying and the peer group: a review. Aggression and Violent Behavior. 2010;15:112–120. doi: 10.1016/j.avb.2009.08.007. [Cross Ref]
52. Durlak JA, Weissberg RP, Dymnicki AB, Taylor RD, Schellinger KB. The impact of enhancing students' social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev. 2011;82:405–432. doi: 10.1111/j.1467-8624.2010.01564.x. [PubMed] [Cross Ref]
53. Espelage DL, Rose CA, Polanin JR. Social-emotional learning program to reduce bullying, fighting, and victimization among middle school students with disabilities. Remedial and Special Education. 2015 doi: 10.1177/0741932514564564. [Cross Ref]
54. Rodgers A, Corbett T, Bramley D, Riddell T, Wills M, Lin R, Jones M. Do u smoke after txt? Results of a randomised trial of smoking cessation using mobile phone text messaging. Tob Control. 2005;14:255–261. doi: 10.1136/tc.2005.011577. http://tobaccocontrol.bmj.com/cgi/pmidlookup?view=long&pmid=16046689. [PMC free article] [PubMed] [Cross Ref]
55. Ybarra ML, Holtrop JS, Prescott TL, Rahbar MH, Strong D. Pilot RCT results of Stop My Smoking USA: a text messaging-based smoking cessation program for young adults. Nicotine Tob Res. 2013;15:1388–1399. doi: 10.1093/ntr/nts339. http://europepmc.org/abstract/MED/23348969. [PMC free article] [PubMed] [Cross Ref]
56. Ybarra ML, Prescott TL, Philips GL, Bull SS, Parsons JT, Mustanski B. Iteratively developing an mHealth HIV prevention program for sexual minority adolescent men. AIDS Behav. 2015 doi: 10.1007/s10461-015-1146-3. [PMC free article] [PubMed] [Cross Ref]
57. Waterlander W, Whittaker R, McRobbie H, Dorey E, Ball K, Maddison R, Myers SK, Crawford D, Jiang Y, Gu Y, Michie J, Ni MC. Development of an evidence-based mHealth weight management program using a formative research process. JMIR Mhealth Uhealth. 2014;2:e18. doi: 10.2196/mhealth.2850. http://mhealth.jmir.org/2014/3/e18/ [PMC free article] [PubMed] [Cross Ref]
58. Whittaker R, Merry S, Dorey E, Maddison R. A development and evaluation process for mHealth interventions: examples from New Zealand. J Health Commun. 2012;17 Suppl 1:11–21. doi: 10.1080/10810730.2011.649103. [PubMed] [Cross Ref]
59. Ybarra ML, Prescott TL, Holtrop JS. Steps in tailoring a text messaging-based smoking cessation program for young adults. J Health Commun. 2014;19:1393–1407. doi: 10.1080/10810730.2014.901441. [PubMed] [Cross Ref]
60. Ybarra ML, Holtrop JS, Bağci Bosi T, Emri S. Design considerations in developing a text messaging program aimed at smoking cessation. J Med Internet Res. 2012;14:e103. doi: 10.2196/jmir.2061. http://www.jmir.org/2012/4/e103/ [PMC free article] [PubMed] [Cross Ref]
61. Center for Innovative Public Health Research San Clemente, CA. 2015. [2016-04-22]. BullyDown http://innovativepublichealth.org/projects/bullydown-b/ webcite.
62. Mustanski B, DuBois LZ, Prescott TL, Ybarra ML. A mixed-methods study of condom use and decision making among adolescent gay and bisexual males. AIDS Behav. 2014;18:1955–1969. doi: 10.1007/s10461-014-0810-3. [PMC free article] [PubMed] [Cross Ref]
63. Thompson D, Cantu D, Bhatt R, Baranowski T, Rodgers W, Jago R, Anderson B, Liu Y, Mendoza JA, Tapia R, Buday R. Texting to increase physical activity among teenagers (TXT Me!): rationale, design, and methods proposal. JMIR Res Protoc. 2014;3:e14. doi: 10.2196/resprot.3074. http://www.researchprotocols.org/2014/1/e14/ [PMC free article] [PubMed] [Cross Ref]
64. Devine S, Bull S, Dreisbach S, Shlay J. Enhancing a teen pregnancy prevention program with text messaging: engaging minority youth to develop TOP ® Plus Text. J Adolesc Health. 2014;54(3 Suppl):S78–S83. doi: 10.1016/j.jadohealth.2013.12.005. http://linkinghub.elsevier.com/retrieve/pii/S1054-139X(13)00799-4. [PubMed] [Cross Ref]
65. Amon KL, Campbell AJ, Hawke C, Steinbeck K. Facebook as a recruitment tool for adolescent health research: a systematic review. Acad Pediatr. 2014;14:439–447.e4. doi: 10.1016/j.acap.2014.05.049. [PubMed] [Cross Ref]

Articles from JMIR mHealth and uHealth are provided here courtesy of JMIR Publications Inc.