A total of 39 afterschool programs from Columbia and Lexington SC and Omaha NE serving a total of 2073 children (kindergarten to 5th grade) with a 1:12 staff:student ratio took part in the evaluation of the instrument. Afterschool program sites ranged in organizations (e.g., YMCA, Boys and Girls Club, individually operated) and locations (i.e., school-based, faith-based, community-based). The average duration of the afterschool program was 3 hours per day. All procedures were approved by the IRB at the University of South Carolina and the University of Nebraska Medical Center.
The HAAND was developed to assess the physical activity and nutritional environment of afterschool programs. Development of the HAAND included an extensive review of existing physical activity and nutrition environmental-quality-rating scales in both childcare and afterschool program settings,9,10,15,17
a review of standards and policies from state and national organizations related to promoting physical activity and nutrition,1,4–6,22,23
and input from individuals with expertise in the field of afterschool child care. The final items included in the HAAND were aligned with existing recommendations, accreditations, and policies from the Council on Accreditation (COA),11
the School-Age Care Environment Rating Scale (SACERS),10
the New York State Afterschool Network (NYSAN) Program Quality Self-Assessment Tool,9
and recently endorsed physical activity and nutrition standards from the National Afterschool Association.6
The HAAND was designed to be collected via direct observation, a brief interview with the afterschool program leader/site director, and written document review during a single day’s visit at an afterschool program. This is consistent with single-day evaluations used to evaluate child care centers and those for school-aged children for accreditation purposes (ncchildcaresearch.dhhs.state.nc.us and www.coastandards.org
The assessment of the afterschool program environment was divided into three sections (revised HAAND tool is available in Appendix A
, available at www.ajpmonline.org
). The first section collects basic information regarding the program name, location (e.g., school, faith or community), weather condition during visit, the number of children attending the program, and the number of staff present the day of the visit. Included in this section are four questions related to the type of snack served at the day of the visit, whether children brought foods or beverages outside (i.e., foods not provided by the afterschool program), whether staff were observed eating or drinking foods in front of children that were not available to the children (e.g., fast food, soda), and the total time allocated for physical activity opportunities in the afterschool program schedule.
The next two sections of the HAAND consist of two separate scales: the Healthy Afterschool Program Index for Physical Activity (HAPI-PA) and nutrition (HAPI-N). These scales were based on a rubric scoring procedure with scores for each item ranging from 0 to a maximum of 4 tables (items operational definitions are given in Appendixes B and C
, available online at www.ajpmonline.org
). The items on the scales were derived from the quality-rating scales, policy/standards documents, and afterschool program professionals (a table showing the alignment of HAAND domain/item with existing standards is available in Appendix D
, available online at www.ajpmonline.org
For the HAPI-PA, a total of six categories consisting of ten total items were developed and for the HAPI-N, a total of seven categories, consisting of 11 items were developed (original HAAND tool). The total score for the HAPI-PA ranged from 0 to 23, whereas the total score for the HAPI-N ranged from 0 to 30. Consistent with state quality-rating scales for the child care setting,9–11
the total score for each scale is presented as either a continuous measure (e.g., 0–23 or 0–30) or presented as an ordinal rating based on a star system, where higher scores indicate a more supportive environment. The stars ratings for the HAPI-PA were 1 to 5 = 1 star, 6 to 10 = 2 stars, 11–15 = 3 stars, 16–20 = 4 stars, and 21–23 = 5 stars, while the star ratings for the HAPI-N were 1 to 6 = 1 star, 7 to 13 = 2 stars, 14–20 = 3 stars, 21–27 = 4 stars, and 28–30 = 5 stars.
Prior to data collection at all the afterschool program sites, the HAAND was pilot tested by two trained observers in an afterschool program to determine whether items were clear, understandable, and able to be answered by observation, interview, and/or document review. After this, revisions were made to clarify questions and items. Once the HAAND was finalized, all field observers (n=7) attended a 1-hour training session that included a review of the HAAND items and scoring system.
All 39 afterschool programs were visited once by at least one trained research assistant. Visits were scheduled with the program unit director at each site, and data were collected in the participating afterschool programs between fall 2010 and spring 2011. The document review involved examining the afterschool program schedule, parent and staff handbooks, physical activity and nutritional training documents (if available), curricula, and policy documents (if available). All the domains were covered during the personal interview. Information from the personal interview, observation, and review documents were used to score HAPI-PA and HAPI-N sections of the HAAND.
Inter-rater reliability testing was conducted on the HAAND to assess the ability of the instrument to yield consistent results across multiple raters. Concurrent HAAND data collection was carried out by two field observers at 20 of the 39 afterschool programs (20 pairs of observations, 51% of afterschool programs). Program unit directors were interviewed by one field observer while responses were recorded by both field observers present during the site visits. After the completion of the interview and direct observation, each field observer reviewed available documents from each site and independently completed the HAAND.
To assess the construct validity of the HAPI-PA scale as a tool to characterize the physical activity environment in afterschool programs, the HAPI-PA items scores were compared to pedometer-determined physical activity (i.e., step counts) collected in a subsample of 934 children attending 25 afterschool programs.18
The pedometer data were collected during the program time over a period of 4 days during the same time period the HAAND data were collected. Pedometer-determined physical activity was measured using the Walk4Life MVPa pedometer, following established protocol in afterschool programs18,24,25
. The Walk4Life pedometer has acceptable reliability and validity in children26,27
. Pedometer-determined step-counts data were collected over 4 consecutive days, Monday to Thursday, with each child having an opportunity to wear a pedometer for a total of 4 days. Valid pedometer data were defined as total time of attendances 60 minutes per day and a minimum of 500 steps per day.24
All 25 sites had data for all 4 days.
To assess the HAPI-N scale validity, observed snack data for the day of the site visit were used to confirm the snack menu for that week. For construct validity assessment, item scores for policy, child feedback, staff training amount and quality, parent workshops, curricula, and evaluation were compared against the mean number of times that FV and whole grains were served in the program per week collected via a program’s snack menu. HAPI-N items related to sugar-sweetened beverages and vending machine access were not compared to FV or whole grain servings per week.
Data were analyzed in summer 2011. Inter-rater reliability was calculated to assess overall agreement for each item on the HAAND using the kappa statistic28
and percent agreement.29
To determine the validity of the HAPI-PA scale, means and SDs were calculated on data from 25 afterschool programs where both HAAND and pedometer steps were collected for boys and girls. Stratified analysis was conducted and comparisons were made among responses on each item on the HAPI-PA (treated as dummy variables with the lowest possible score serving as the reference group) with pedometer step count for boys and girls, separately.
All analyses used a three-level random intercept model (days nested within children nested within afterschool programs). For the HAPI-N scale, means and SDs were calculated on data from 39 afterschool programs and comparisons were made among responses using one-way ANOVAs with each items’ response categories on the HAPI-N as the independent variable and the average number of times that FV and whole grains were served in the program per week as the dependent variable. Significance was set at p<0.01.