|Home | About | Journals | Submit | Contact Us | Français|
Patient-centered evaluation is a critical but often overlooked component of intervention research. The Rural Breast Cancer Survivors (RBCS) Intervention is a survivorship education and support intervention designed for rural breast cancer survivors. Here, we describe evaluation of the RBCS Intervention from the participants’ experience.
Three hundred sixty-four breast cancer survivors participating in the RBCS Intervention were asked to complete the RBCS Evaluation Survey that consisted of 6 quantitative statements and 12 qualitative questions via mail.
Two hundred twenty-one participants (61 %) returned the RBCS Evaluation Survey. All 221 completed the qualitative component, and 91 (25 %) completed the quantitative. Overall, participants indicated that the education materials were easy to understand and addressed their quality of life concerns. Majority (94 %) indicated that the survivorship information was helpful in making health decisions and communicating concerns to family (87 % strongly agree/agree) and oncology team (88 % strongly agree/agree). Only 66 % indicated that the survivorship educational materials were provided at an appropriate time after completion of primary breast cancer treatment. Qualitative data suggested that participants had positive perceptions and experiences and found the RBCS Intervention helpful. One-on-one interaction with the intervention nurses was the single most highly valued aspect.
Overall, the RBCS Intervention was helpful. Education materials addressed quality of life concerns. Breast cancer survivors developed skills in communicating with their family and oncology team. Intervention nurses were identified as the most valuable aspect of the program. Suggestions include earlier timing of survivorship education and support, and adaptation using social media access to serve other rural survivors.
Patient-centered evaluations, using both quantitative and qualitative data, provide enriched understanding of evidence-based survivorship interventions and should be considered a standard for future work.
A patient-centered approach to outcome evaluation is critical for understanding program experiences and outcomes from the participant perspective and for improving dissemination of known and effective interventions [1, 2]. A common approach to program evaluation has been survey with closed-ended questions . Yet, there are known drawbacks to this approach [4, 5]. For example, the restricted response format limits the participants’ ability to express any negative experiences so that any underlying perceptions and experiences may not be revealed [4, 5]. It may be advantageous and important to add a qualitative approach to explore participant-perceived outcomes, both the positive and negative experiences, and patient perceptions of ways to improve the program for the future [2, 3]. Research has shown that data collected by open-ended questions generate more detailed and precise information about participants’ opinions of the program in terms of good and bad facets compared to data collected by closed-ended questions [3, 6]. Thus, patient-centered evaluations can be used to better understand how participants’ view an intervention’s strengths, weaknesses, and problems, and to evaluate the value to patients (e.g., helpfulness and usefulness) .
Patient-centered evaluations have been used in self-management, nutrition, and youth development education programs [2, 3, 6, 7]. Here we report our participant-centered evaluation approach, where both quantitative and qualitative data were collected to evaluate participant satisfaction and usefulness of the Rural Breast Cancer Survivors (RBCS) Intervention. The purpose of this paper is to assess the participants’ overall perceptions and experiences with the RBCS Intervention and the RBCS written educational materials for the purpose of refining the intervention and the selected components for future dissemination.
The RBCS is a telephone-mediated intervention adapted for underserved rural breast cancer survivors having limited geographic access to survivorship education and support. The RBCS Intervention was modeled on the Breast Cancer Education Intervention (BCEi), a randomized trial demonstrating efficacious survivorship educational and support interventions in breast cancer survivors . The program emphasized patient-centered self-management of treatment late effects, psychosocial concerns, and work and insurance challenges . The BCEi was cited in a 2012 Cochrane review demonstrating that education combined with support improved patient-reported outcomes among breast cancer survivors .
The BCEi was later evaluated by an expert panel of the Research-Tested Intervention Programs at the National Cancer Institute  who judged the BCEi using Research-Tested Intervention Programs (RTIP) indicators of research integrity, intervention impact, and dissemination capability . In addition, the panel evaluated the BCEi using RE-AIM principles of reach, effectiveness, adoption, and implementation . The panel concluded that the BCEi was an effective, evidence-based intervention with high dissemination capability. Building on this evidence with the goal of extending dissemination and reach of the BCEi to rural underserved breast cancer survivors, the authors conducted a preliminary evaluation of the BCEi for rural survivors  and subsequently modified the program using telephone delivery instead of face-to-face intervention. The telephone-delivered program, called the Rural Breast Cancer Survivors Intervention (RBCS), was a population-based randomized trial investigating whether there were differences between the Early Education and Support Intervention or the Support and Delayed Education Intervention.
The RBCS was approved by the University of Alabama at Birmingham Institutional Review Board and the Florida Department of Health Institutional Review Board. The present study consisted of participants’ evaluation of the RBCS with respect to helpfulness, usefulness, and satisfaction after completing the primary study which required 12 months of participation.
Women diagnosed with stage 0–III breast cancer, residing in rural Florida, within the first 3 years following diagnosis of breast cancer, at least 6 months following completion of primary treatment, at least 21 years of age, and with telephone or cell phone access were eligible for the RBCS. Rural breast cancer survivors receiving anti-hormonal therapy or anti-HER2 treatment at study entry were also eligible. To be a rural resident, women either resided in 1 of 33 rural counties designated by Florida statute  or lived in a rural pocket of one of 34 Florida urban counties having an Index of Research Access (IRA) score equal to or greater than 4 . The IRA score is a measure of rurality and was preferred over other measures because it provided a granular determination of rurality in urban counties that is not captured by zip code or county residence alone .
The authors used approved recruitment procedures designated by the State of Florida Cancer Data Registry where contact was through letter of study invitation and/or telephone call. Breast cancer survivors could respond via mail or telephone call to the research office regarding interest or disinterest in participation. Those who agreed to participate provided written informed consent. Those who declined were removed from the recruitment list to prevent future contact. After signed informed consent, participants in the RBCS were randomly assignment to either the Early Education and Support Intervention or the Support and Delayed Education Intervention group.
A total of 432 participants enrolled in the parent RBCS study. For the present analysis, data included 221 participants who returned the RBCS Evaluation Surveys.
The RBCS Intervention components and delivery were described in detail elsewhere [15, 16]. Briefly, there were four educational intervention components: one Intake Assessment (IA), three Education calls (ED), one Follow-Up Education call (FUE), and six Support calls (SUP). The intervention groups differed in the timing of the three ED, the Early Education group received ED in month 1, and the Delayed Education group received ED in month 7. The RBCS Intervention was delivered by trained oncology research nurses [15, 16].
Since this was a telephone-delivered intervention in which the participant and interventionist had never met in person, the purpose of the IA was to establish contact and rapport with participants. The three ED were delivered using a quality of life framework where physical, psychological, and psychosocial well-being, and healthy self-management tips were discussed. In ED #1, interventionists discussed physical effects of treatment ranging from cancer-related fatigue, lymphedema, pain, and cognitive changes. In ED #2, interventionists described strategies to promote healthy lifestyle behaviors including physical activity and healthy nutrition, changes in family and social relationships, menopausal changes, and work, financial, and/or insurance concerns. In ED #3, interventionists explored psychological late effects including anxiety and depression, fear of recurrence, and spiritual changes that can often occur up to several years in survivorship. The FUE helped the interventionists to evaluate participants’ learning about survivorship self-management and to reinforce the need for regular cancer surveillance follow-up, health and wellness activities, and symptom management. The fourth component was support calls (SUP) that were scheduled monthly to evaluate how participants were self-managing and to support them in adopting or modifying health behaviors.
Learning activities and self-management were supplemented by a 140-page RBCS Education Binder and 38 Tip sheets. The education materials were organized into six sections and contained written information corresponding to the three ED sessions described above with two sections discussed at each ED session. The Tip sheets are short, one-page, bulleted items that provide a quick snapshot of self-management behaviors. All written materials were at the sixth grade reading level. The written education and Tip sheets are publicly available at the National Cancer Institute’s Research-tested Intervention Programs (RTIPs) website at http://rtips.cancer.gov/rtips/programDetails.do?programId=1416306
For this analysis, the RBCS Evaluation Survey was developed by the authors and contained quantitative and qualitative components. The quantitative component consisted of six statements addressing participant satisfaction with the written RBCS Education binder with regards to the participants’ overall understanding of the information, quality of life concerns, helpfulness in decision-making, helpfulness in communicating with family and oncology team, and whether the timing of the education materials was appropriate. Responses to each of the six statements were based on a four-point Likert-type scale ranging from strongly agree to strongly disagree. See Table 1.
The RBCS Evaluation Survey qualitative component consisted of 12 open-ended questions about participants’ perceptions of the RBCS Intervention with respect to their overall impression of the RBCS, whether the RBCS met their expectations, helpfulness, usefulness, and sufficiency of the different RBCS Intervention components, recommended changes, and suggestions for improvement. See Table 2.
The RBCS Evaluation Survey was mailed to the 364 of the 432 participants who completed 12 months of study participation in the RBCS Intervention. No identifiers or demographic information was included in the RBCS Evaluation Survey; thus, responses returned were anonymous.
Quantitative analyses of responses to the six questions were analyzed using descriptive statistics .
Qualitative thematic content analysis was used to analyze participant program experiences . The qualitative data were analyzed in three stages. Initially, an investigator (YS) read through all the data and coded the data to identify categories and emerging themes and independently coded the responses into the identified categories and created new categories, if necessary. Lastly, the categories were collapsed into three themes, and comparisons were made between the investigators to reach a consensus on the data within categories and themes. This iterative process continued until the investigators agreed on the coding scheme and the frequency a code appeared in the data.
Two hundred twenty-one of 364 participants (61 %) returned the RBCS Evaluation Survey via mail. All 221 answered the qualitative component, and 91 (25 %) answered the quantitative component. Since the RBCS Evaluation Survey did not contain demographic data, the specific characteristics of this sample could not be determined. However, in the parent RBCS, the sample was predominately Caucasian, with a mean age of 63.1 years, married/partnered, and educated at the trade school/some college level. More than 30 % had household incomes <$30,000 [15, 19].
Overall, participants expressed high satisfaction with the RBCS survivorship education materials (see Table 1). All participants indicated that the education materials were easy to understand and addressed their specific quality of life concerns. The majority (94 %) indicated that the survivorship information was helpful in making health decisions. The majority of participants found that the education helped them communicate concerns to their family (87 % strongly agree/agree) and oncology team (88 % strongly agree/agree). Only 66 % indicated that the survivorship educational materials were provided at an appropriate time after completion of primary breast cancer treatment, strongly suggesting that education was preferred earlier than later.
Analysis of the qualitative data suggested that most participants had positive perceptions and experiences and found the RBCS Intervention very helpful. A total of 1679 comments were coded from the 12 open-ended questions. The comments were grouped into three major themes: (1) perceptions and experiences, (2) perceived helpfulness, and (3) suggestions for future improvements. Categories within each theme are described further with illustrative examples of participant comments. Table 3 illustrates the frequency of the themes and categories.
Thirty-four percent of the responses concerned participants’ perceptions and experiences. Overall, they found the RBCS Intervention to be comprehensive, informative, and helpful. They enjoyed participating and would recommend the RBCS Intervention to others including family and friends. Typical quotes include the following: “I asked my doctors better questions and feel I received more info from them than I would have without your information.” “The RBCS Program addressed so many issues of post treatment survivors that most people are unaware of.”
Several participants did not know what to expect from the RBCS Intervention, but after completion, they indicated that the intervention exceeded their expectations. “The study exceeded my expectations because I did not foresee all the education, counseling, and self-knowledge I would receive.”
Some survivors also communicated an altruistic sense of wanting to help others as another reason for participating in the RBCS Intervention. “I did not have the problems most women have with breast cancer. I did this study for the future of my daughters and problems they might face.”
Thirty-six percent of responses concerned perceptions of helpfulness of the RBCS Intervention. Participants identified that the one-on-one interaction with their nurse interventionist was the most helpful aspect of the RBCS and the most frequent comment, accounting for 24 % of the total comments. In general, the participants felt that the intervention nurses were compassionate, supportive, and highly knowledgeable about post-treatment survivorship issues. “I liked talking to a person who was very objective. I was able to put the whole ‘breast cancer experience’ aside and see how it had affected me and what could be done to bring me back in focus.”
“I love to say the best part of the study was the people running the study, kind, and caring, knowledgeable helpful and just plain fun to talk to. Something we all need at times, especially the bad times.”
“I do a lot of research (internet) on my own and read a lot of professional journals on the subject of cancer and its treatment side-effects. What was lacking was my inability to “talk” to someone and tell her my intimate feelings and problems. [Interventionist nurse name] was able to bridge the gap with her one-on-one, caring conversations.”
While specific questions about the RBCS Education materials contained in the binder and tip sheets are described in Table 1, 8 % of comments in the qualitative component also concerned survivorship education materials. Participants indicated that the survivorship education and self-management were important. Three topics identified as most helpful were managing fatigue, menopausal symptoms, and maintaining health. Other outcomes they gained by the RBCS Intervention were enhanced self-awareness and knowledge of survivorship issues. Herewith are some examples: “I was having trouble with fatigue. I had returned to work and was thinking it might have been too soon or I was getting sick again because I was so tired. Imagine my surprise when the 1st chapter was dealing with fatigue!”
“The ‘New You/Maintaining Health’ was great initially and even more valuable when I reread the information 10 months later and realized that I hadn’t followed through with, like eating a proper diet.”
“I was able to see I could give myself permission to do what I could and not do what I felt I couldn’t handle.”
“[Knowing about survivorship issues] helped me understand that what I was dealing with and feeling was normal.”
Furthermore, 4 % of participants indicated that the tip sheets provided helpful information that they will use for future reference. “Tip sheet information helps with after the chemo/radiation- what to expect and how to deal with the challenges- very informative a great deal of help to me!”
“Throughout the year, I found myself checking out the tip sheets for information and I expect to keep doing so from time to time.”
Thirty percent of comments were related to suggestions for future improvement. While 19 % of comments recommended no changes to the RBCS Intervention, 11 % of comments concerned two major improvement suggestions. First, the future RBCS Intervention should include additional discussion on drug side effects and symptoms accounting for 6 % of comments. “I think it may be helpful to know what percentage of other patients experience side-affects so we can see the commonality of certain physical problems such as fatigue-while other physical problems may be less common. I wonder what other types of after-treatment symptoms where being experienced from other cancer survivors.”
Second, participants suggested that the RBCS Intervention should be delivered earlier during active treatment rather than post-treatment survivorship accounting for 5 % of comments. “Try to get to women soon after their treatment. So they could get through these times and changes in their body and understand what is going on with them.”
“I think it would be good to start this when people are first diagnosed. The educational book would have helped alleviate fears when I was first diagnosed. I did search for answers, but had been warned that not everything on the internet is good to get info from. Since you had recommended sites, I could have looked there knowing the info was good.”
Several lessons learned from this study deserve mention. First, this study underscores the value and importance of patient-centered evaluation of any program, particularly intervention studies. We found that the combination of quantitative and qualitative components to the RBCS Evaluation Survey were enriching and revealing. The 1679 qualitative comments provided a broad understanding of participants’ perceptions, experiences, and perceived helpfulness of the RBCS Intervention. In addition, they added immense value suggesting changes for future improvement. All too often in intervention studies, researchers either forget or neglect to ask the recipient of the research the basic question: “How well did this intervention help you?” Findings from this study stress the importance of incorporating both qualitative comments and quantitative evaluation responses in future studies of cancer survivorship interventions.
Second, the telephone-delivered survivorship intervention allowed the authors to reach rural breast cancer survivors in 63 of the 67 counties in the State of Florida, areas where electronic access was not well received. Thus, reach of the RBCS using low-technology telephone was highly efficient. However, during the 5 years of this study, participants became increasingly adept with use of social media that allowed them to remain engaged at a distance. Yet, they also underscored the vital importance of the human voice located hundreds of miles away to convey strength, compassion, and caring for them as rural breast cancer survivors.
Our findings showed that the telephone is an efficient intervention delivery mode well suited to reach large underserved populations. Further, our findings support other studies of telephone-based interventions including Latina breast cancer survivors with depressive symptoms , breast cancer survivors and partners [21–23], and across other survivors along the breast cancer trajectory [24–27]. The benefit of telephone-delivered intervention in multilingual countries around the globe is not yet known.
Third, the one-on-one interaction with nurse interventionists was the single most highly valued aspect of the RBCS Intervention. Engagement with the nurses helped rural breast cancer survivors transition into survivorship. Also, hearing the same voice of the interventionist on the telephone to educate and support participants throughout the entire 12 months of study participation had lasting value. Intervention nurses bridged the gap between end of treatment and moving into survivorship for the rurally underserved. They could be counted on to call on a monthly basis and evaluate how well participants were managing, and to hear about daily aspects of the lives of rural breast cancer survivors during their monthly check calls.
Fourth, enjoyment was an important attribute of participant satisfaction among this population of rural breast cancer survivors. Although cited in 2 % of comments, the authors were surprised to find perceptions of enjoyment. Likely, when participants enjoyed the interaction, they expressed high satisfaction. On the other hand, from a research and clinical perspective, developers of interventions are most concerned about accurate program information and helpfulness, not necessarily enjoyment. The concept of enjoyment among patients is rare in the health care literature. This concept is usually associated with other industries such as entertainment . Future evaluation of patient-centered outcomes may consider the concept of enjoyment and how this may be similar to engagement.
Fifth, participants commented that the RBCS was educationally useful. Particularly, participants cited the practical and applicable strategies on self-management. These findings are important for future work in survivorship self-care management where survivors will increasingly be expected to shoulder more of the responsibility for their own self-maintenance and adherence to follow up cancer recommendations [29, 30].
Sixth, the timing of intervention delivery is vital. The quantitative responses showing that 34 % disagree/strongly disagree with the timing of the intervention were congruent with the 5 % of qualitative comments that survivorship education and self-management be delivered sooner rather than later. Since the RBCS Intervention used a population-based approach to recruit rural breast cancer survivors, the delayed timing of the intervention was an artifact of retrospective data provided by Cancer Registry, thus the lag in delivering the intervention. Recent data about survivorship interventions indicate that receipt of survivorship self-management information is ideally accomplished “sooner rather than later” . In future dissemination studies, telephone-based survivorship information can be ideally accomplished with cancer survivorship care planning while survivors are in active treatment .
Finally, future dissemination and reach strategies of the RBCS are to test new modalities for intervention delivery. For example, text messaging was suggested by a participant to “be able to TEXT your nurse.” Participants also suggested using other technology such as electronic group sessions. “I would love to see a web site where survivors can talk with each other about the struggles” and “Maybe an interactive website for those that are computer savvy.”
Several study limitations are noted. First, a separate demographic questionnaire was not included with the RBCS Evaluation Survey. Thus, we were not able to evaluate whether there were sociodemographic or cancer treatment differences between participants in the parent RBCS and those returning the RBCS Evaluation Survey. Second, the survey was mailed to participants who completed the RBCS Intervention or who withdrew either verbally or in writing. Participants lost to follow-up who did not indicate a desire to withdraw were not sent the RBCS Evaluation Survey. Thus, those who withdrew from the program without notifying the PI may not have been satisfied with the program and were not given the opportunity to provide feedback about the RBCS. Third, participants were not asked how specific components of the RBCS Intervention could be translated to social media technology. While a few participants mentioned the use of technology, a specific question on social media and technology could have provided more data for broader adaptation, dissemination, and implementation of the RBCS.
Overall, participants were satisfied with the RBCS Intervention. More participants responded to the qualitative rather than the quantitative evaluation components. Future behavioral and quality of life studies of cancer survivors would benefit from patient-centered approaches not only at the outset before implementation of a survivorship program but also, more importantly, to overall program evaluation which can lead to further refinement and improvement of future survivorship programs. Participants who completed the RBCS Evaluation Survey provided excellent suggestions for modifying evidence-based intervention for future rural breast cancer survivors. The study revealed the potential critical importance of early and persistent human contact from diagnosis to life after treatment.
The authors would like to thank our breast cancer survivor participants who gave generously of their time.
Funding source Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA120638. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Compliance with ethical standards
Ethical approval All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Disclosure The Florida cancer incidence data used in this report were collected by the Florida Cancer Data System under contract with the Department of Health (DOH). The views expressed herein are solely those of the author(s) and do not necessarily reflect those of the contractor or DOH.
Conflict of interest The authors declare that they have no conflict of interest.
Informed consent Informed consent was obtained from all individual participants included in the study.