The evaluation study comprised of two key components. In order to evaluate the overall effectiveness of the intervention, a randomised controlled design was employed. Randomised controlled trials (RCTs) are generally considered the most reliable means of assessing intervention efficacy (e.g., see Campbell et al [29
]). Since public health interventions, such as the school breakfast initiative, tend to act at multiple levels and through multiple channels [30
] implementation of a traditional RCT is difficult. It is likely that these methodological difficulties have contributed to the absence of good evidence for the efficacy of school breakfast programmes and other public health interventions. However, recent developments in research methodology, including the use of cluster RCTs and mixed methodologies have the potential to provide unbiased estimates of the effectiveness of such interventions as well as identifying other factors that are more variable such as intervention delivery, context and support [31
As the intervention was implemented at the school-level, randomisation of individual students to control or intervention groups was not practicable. As such the evaluation adopted a cluster randomised design with school as the unit of randomisation. A summary of this study design is presented in Figure . In line with the aforementioned development of mixed-methodology approaches to evaluation, a nested qualitative process evaluation was included to address issues concerning the context and implementation of the initiative. In this way, the evaluation not only addressed the question, 'Does it work?', but also considered 'What works?', 'For whom?' and 'Under what circumstances?' [32
Timetable for evaluation of the Welsh Assembly Government's Free School Breakfast Initiative.
The Primary School Free Breakfast Intervention
The intervention provided a school-based breakfast before the commencement of classes, without any cost being borne by parents. The aim of the intervention was not only to encourage breakfast consumption per se, but to improve the nutritional quality of children's breakfasts. Hence a particular focus was placed upon limiting available food choices to breakfast items considered to be healthful. Provision was therefore offered from four food types: non-sugar coated cereals, bread, milk products and fruits, in addition to drinks. Schools were provided with guidelines regarding how the scheme should be run, in terms of staff to student ratios and the food to be provided, but were given a reasonable degree of autonomy in the operation of the scheme.
The scheme was rolled out in two distinct phases by the Welsh Assembly Government, with initial priority given to schools in more deprived areas of Wales. As such, schools classified as 'Communities First' schools were eligible to receive the intervention initially before the scheme was made available to all schools the following year. At the time of writing, information about the scheme can be found on the Welsh Assembly Government's website [33
Recruitment and randomisation of schools
Phase one – Communities First schools
In autumn 2004, head teachers of infant, junior and primary schools located in 'Communities First' (i.e. deprived) areas in nine local education authorities (LEAs) in north, south and west Wales were invited to participate in the evaluation. Schools were offered £250 to compensate for additional teacher time and disruption to school activities. One hundred and fifty-two schools were approached and 58 schools agreed to participate. Reasons for non-participation were generally related to the running of the breakfast scheme itself (worries about implementing the new initiative and bringing forward the start date) rather than any concerns over data collection burden. Those that agreed to participate were randomised to the intervention or control condition using strata defined by LEA, school size, free school meal entitlement and Welsh language medium.
Phase two – non-Communities First schools
Head teachers of infant, junior and primary schools located in 'Non-Communities First' (i.e. more affluent) areas, in the same nine LEAs were invited to participate in the evaluation. As before, schools were offered £250 to compensate for additional teacher time and disruption to school activities. This time, 456 schools were approached and 53 schools agreed to participate. Again, reasons for non-participation were generally related to fears about relinquishing control over the start date of the scheme, although a number of differences in reasons for non-participation unique to this phase of the study were identified (see response biases section of Discussion). Those who agreed to participate were randomised to intervention or control following the same methods of stratification as described above.
For both phases, schools in the intervention group were asked to set up a breakfast scheme, following the guidance issued by the Welsh Assembly Government, after baseline measures had been collected. Schools in the control condition were asked to refrain from setting up a breakfast scheme until after the 12 month measures had been collected.
In each of these 111 schools, following the timetables set out in Figure , one class of Year 5 children (9–10 years) and one class of Year 6 children (10–11 years) were randomly selected to complete class based measures. From this sample, 5 children from each year (i.e. 10 in total) were randomly selected to be assessed by teachers (see below) and 6–9 of these children were randomly selected to participate in individual testing. In addition, for each school 35 children were randomly selected from the full age range, and a questionnaire was sent to their parents. For primary schools, 5 children were sampled from each year group (i.e., reception to year 6) and for junior schools, 8 or 9 students were selected from each year group (i.e., years 3 to 6).
The quantitative measures employed in the evaluation aim to provide an accurate assessment of the impact of the scheme on children's dietary habits, cognitive performance, attitudes and classroom behaviour. As previously discussed, since the intervention is implemented at the school-level, randomisation has to take place at this level. As a consequence, relatively large numbers of participants are needed in order to detect any intervention effect. For such a trial to be feasible it is therefore important that the outcome measures employed are relatively quick, cost-effective and easy to implement. In an attempt to address the inevitable tension between measurement precision and sample size/response bias (i.e., brief measures, whilst cost effective and perhaps encouraging higher response rates, are likely to suffer from a higher degree of measurement error) our evaluation incorporated two levels of assessment. For dietary measurement, the study first used a less sensitive dietary recall questionnaire administered simultaneously to the whole class. Secondly, a subsample of children also participated in a validated, and much more time consuming, one-to-one dietary interview procedure.
The following standardised, previously validated measures were used:
Classroom administered cognitive tests were used to assess episodic memory, working memory, sustained attention and psychomotor speed [34
]. Individually administered, computerised cognitive tests were used to assess sustained attention, selective attention, simple reaction time and choice reaction time. The most consistent effects of breakfast upon cognition, in experimental conditions, have previously been observed for episodic memory [35
] which was therefore selected as the primary outcome in terms of cognitive function.
Strengths and Difficulties Questionnaire
The Strengths and Difficulties Questionnaire [39
] was completed by teachers to assess children's classroom behaviour. This was a relatively brief, but well-established measure that assessed five dimensions of behaviour: hyperactivity, emotional symptoms, conduct problems, peer problems and pro-social behaviour. Hyperactivity in particular, is likely to be influenced by breakfast due to its relation to on-task behaviour [40
]. Therefore this sub-scale will be analysed as a secondary outcome, with the global total difficulties scale analysed as a tertiary outcome.
Dietary recall interview
Individually administered dietary recall interviews were used to provide a more accurate estimate of the impact of the initiative on children's diets. These interviews were conducted using a standardised protocol based on that employed by Lytle et al. [41
Due to a shortage of previously validated measures the following measures were developed and validated for use in the present study [42
Dietary recall questionnaire
Children were asked to list all foods and drinks consumed at chronologically ordered time points throughout the day (e.g., at home before school, on the way to school, at school before class started). Details of breakfast on the day of reporting (i.e., any foods consumed before the start of classes) were collected first, followed by details of the previous day's intake [42
]. Primary measures from this questionnaire are the number of healthy food items (i.e., fruit, bread, cereal and milk products) and number of unhealthy food items consumed at breakfast (i.e., sweet items and crisps), and the number of days on which breakfast was consumed in the last two days (i.e., 0, 1 or 2).
Attitudes towards eating breakfast
Attitudes were assessed using a questionnaire containing thirteen statements referring to a variety of domains, such as concentration and behaviour, energy, and the general importance placed on breakfast. Children were asked to indicate the extent to which they agreed or disagreed with each statement by placing a tick in one of 5 boxes (agree a lot/agree a bit/don't agree or disagree/disagree a bit/disagree a lot) [43
The parental questionnaire contained 10 questions designed to assess children's breakfast eating habits. Five of these asked parents how many times on school days their child usually engaged in a particular behaviour (ate breakfast at home, took something from home for breakfast to eat on the way to school or at school before the start of class, took money to buy breakfast on the way to school, ate a breakfast provided by the school, missed breakfast). These were answered by placing a tick in one of 7 boxes (number of days ranging from 0 to 5 or 'Don't know'). Four questions asked parents how many times at the weekend their child usually engaged in a particular behaviour (ate breakfast at home, took something from home for breakfast to eat elsewhere, took money to buy something for breakfast, missed breakfast). These were answered by placing a tick in one of 4 boxes (number of days ranging from 0 to 2 or 'Don't know'). An additional question asked parents to rate the frequency with which they thought their child ate a healthy breakfast. This was answered by placing a tick in one of 5 boxes ranging from 'My child always eats a healthy breakfast' to 'My child rarely eats a healthy breakfast'.
Sample size calculations
Since there are a range of outcome measures, sample size requirements were calculated using effect sizes. Sample size calculations assume an intra-cluster correlation of 0.02, 80% power, and a two-tailed alpha of 0.05. With 111 schools in the trial, for pupil outcomes from the self-complete questionnaire, assuming an average of 50 responses per school, there will be power to detect an effect size of 0.11. For parent reports of breakfasting behaviour, assuming 20 responses per school, there will be power to detect an effect size of 0.15. For pupil outcomes from the dietary recall interviews, there will be power to detect an effect size of 0.2.
Parents were informed of the research in advance by means of a letter and information sheet either posted to them or sent home with children, and were asked to complete a return slip and/or contact the school if they did not wish their child to participate in the study. At each data collection, children were also informed that they were under no obligation to participate. The study received ethical approval from the Cardiff University Social Science Ethics Committee.
Cognitive tests, the attitudes questionnaire and the dietary recall questionnaire were completed between 9 am and 12 pm as supervised classroom exercises with a maximum class size of 40 children. For the attitudes questionnaire, to minimise conferring and ensure that children worked at the same pace and did not distract one another, the researcher read out the statements one by one and children marked their response for each statement after it was read out. For the dietary recall measure, the researcher read out the instructions and asked children to complete the questionnaire independently from one another. Children were asked to put their hands up when they had finished or if they needed help with spelling, or further clarification of questions. Three members of the research team were present to assist children.
As described above, at baseline and 12-month follow-up only, 3–5 children from each year group (e.g. year 5 and 6) were selected to complete individual tests between 12:30 and 3:30 pm (although timetabling restrictions occasionally led to these being conducted as early as 11:00 am). These were conducted on a one-to-one basis with one researcher guiding each child through the computerised cognitive tests and the dietary recall interview.
Schools were divided between key members of the research team, and each key researcher was responsible for all liaison with their assigned schools, and held principal responsibility for data collections. Two trained temporary research assistants were also brought to each school at baseline and 12-month follow-up, to provide general assistance with group testing and to conduct individual testing in the afternoon. For the first follow up, only one assistant was required as only the morning's group testing was carried out. All temporary research assistants were fully trained and monitored by the key researchers in order to maximise standardisation across the trial sites and data collection sweeps.
There was no one primary outcome, but a pre-specified analysis plan was agreed in which the following were identified as primary outcomes: the proportion of students consuming two breakfasts over two days; episodic memory; number of healthy food items consumed at breakfast and number of unhealthy food items consumed at breakfast according to the dietary recall questionnaire. Secondary outcomes were identified as attitudes towards eating breakfast; rest of day healthy food items; rest of day unhealthy food items; scores on the hyperactivity/inattention scale of the strength and difficulties questionnaire, and parental reports of frequency of eating breakfast at home and at school. A number of other outcome measures were a priori defined as tertiary outcomes, including cognitive measures other than episodic memory, the total difficulties score on the strengths and difficulties questionnaire, parental reports of morning routines and child care arrangements before school and variables collected from the individually administered in-depth dietary recall and cognitive measure interviews.
For each outcome variable, the primary analysis is a school-level weighted regression analysis [44
] adjusting for baseline score and the four stratification variables. These primary analyses will be conducted on an intention-to-treat basis, in which each school is coded according to the treatment condition to which it has been randomised. A secondary analysis conducted for all variables will be a per protocol analysis, in which schools are coded according to whether or not a free breakfast scheme was set up prior to outcome measurement.
Sub-group analyses are planned for study phase, socio-economic status and groups defined by consumption of breakfast, healthy and unhealthy food items at baseline. Socio-economic status will initially be defined in terms of a school level variable indicating the proportion of students entitled to free school meals, but through further data linkage it is planned that student level socio-economic measures based on postcode of residence will be available. If this additional data linkage is undertaken, it will also allow the performance of students in Statutory Assessment Tests and public examinations to be included as additional (secondary) outcome variables, although this is not included in the current protocol as it is dependent on ethical approval and funding. Other sub-group analyses may be conducted according to variables identified in the process evaluation, which may include measures on the implementation of the scheme or relating to the school and community context.
Whilst recognising the need to adopt appropriate research designs and to draw on theoretically informed outcome measures to evaluate complex interventions, it is of equal importance to understand processes [45
]. The evaluation therefore incorporated a substantial process element that examined how the initiative had been implemented. This will facilitate the interpretation of outcome effects and add to the understanding of how a major policy initiative is undertaken.
An initial preliminary process evaluation was also completed with schools who began provision of free breakfasts during the first wave roll out in September 2004. This consisted of telephone interviews with the Welsh Assembly Government breakfast team, LEA co-ordinators and questionnaires with school based co-ordinators. In addition, a number of case study schools were selected for an in-depth observational and interview study with teachers, caterers and students. This preliminary study allowed process measures and an appropriate design to be developed and piloted before the main process evaluation was undertaken. Results also formed the basis of a report to the Welsh Assembly Government on the initial implementation with recommendations for policy and practice [46
The main process evaluation consisted of telephone interviews with LEA co-ordinators, a postal questionnaire for school co-ordinators of the scheme and interviews and observation in case study schools. Following lessons learnt from the preliminary study, the process evaluation drew on the framework proposed by Steckler and Linnan [47
] to examine the following areas:
1. Context. Existing dietary health promotion activity to determine the relationship between school climate, implementation and outcomes [48
2. Fidelity and dose. Details of initiative content and a comparison of daily record keeping for the Assembly Government and for guidance documentation.
3. Integration. Details of existing, and any changes to, school policies and the level of integration with curriculum and extra-curricular activities.
4. Recruitment, participation and reach. Approaches to promotion and recruitment, any use of targeted recruitment, level of, and explanation for, participation and non participation.
5. Implementation and sustainability. Barriers and facilitators to implementation, level and type of staff involvement, level and type of parental involvement, in and out of school time involved, direct and indirect financial costs, benefits and costs to school, staff, students and parents.
The case study schools consisted of eight participating schools from each phase of the evaluation that were purposefully selected to explore these issues in greater depth. Schools were chosen so that differences in size of school, local education authority, setting (e.g. urban or rural) and percentage of free school meal entitlement were reflected. School selection was determined by information gathered from the teachers, uptake records and school characteristics. In this way a range of approaches to, and experiences of, implementation were examined. In these schools, semi-structured interviews were conducted with head teachers, teachers, breakfast scheme staff and students attending the scheme. The interviews asked for accounts of any changes to the school, dietary behaviour, school behaviour, attitudes and norms as well as views on, and experiences of, the initiative and its implementation. Breakfast schemes were observed in each of the case study schools with observational records taken of the delivery of the intervention.
Finally a questionnaire was sent to head teachers in all participating schools asking about policies, initiatives and contextual issues that had occurred over the course of collecting the quantitative data. In this way, events that may have affected the outcome data could also be taken account of.
Participation rates and school attrition
Of the schools where baseline data were collected, no schools withdrew from the study prior to final data collection. There were however five schools randomised to the control group who nevertheless set up a free breakfast scheme prior to the 12-month follow up, while ten schools randomised to the intervention group had not set up the scheme within the follow-up period. Thus the per protocol analyses have 61 schools coded as control schools and 50 as intervention.
Participation of students
Response rates for students participating in classroom data collections are presented in Table . Whilst there was some minor variation between measures in terms of responses, with for example some children arriving late in class having missed one measure or having to leave before the final measure had been completed, these differences were negligible. Response rates for cognitive testing ranged from 85.5 to 87.9%, for the attitudes measure from 86.3 to 88.3% and for the dietary recall questionnaire from 86.2 to 88.4%. For ease of reading, detailed response data relating to only the attitudes towards breakfast questionnaire are presented, with the choice of measure entirely arbitrary.
Response rates and reasons for non-participation amongst students within control and intervention schools, at baseline, 4-month and 12-month follow up
For both control and intervention schools, and across all 3 data sweeps, data were collected from more than 85% of those eligible to take part, with the majority of non-participation due to the child's absence from the school on the day of testing. Only very small numbers of children were excluded either by parents prior to data collections or by their decision not to give consent on the day of testing. The variation between the numbers of children sampled at each sweep reflects the clustered nature of the sampling strategy, with repeated cross sections sampled from within each school, varying marginally from one time point to the next due to changing class compositions.
The flow of participants through the study is presented in Figure . The inflow and outflow of participants between the first and 12 month follow ups reflects the fact that the follow-up of each school at 12 months is in effect a repeated cross-section design from the perspective of students. Thus, approximately half of the cohort recruited for the baseline and first follow-up measures are replaced at 12 months by an incoming group of year 5 students. These repeated cross-sections will be used for school-level analyses which are the primary analyses for the trial. Additional research questions may be addressed through analysis of the nested cohort of participants eligible to participate at all 3 time points, of whom there were 1975 who provided data at baseline and one or both of the follow-ups.
Flow of participants through the study.