Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Dev Behav Pediatr. Author manuscript; available in PMC 2011 June 1.
Published in final edited form as:
PMCID: PMC3057358

Improving Access to Care and Clinical Outcome for Pediatric Behavioral Problems: A Randomized Trial of a Nurse-Administered Intervention in Primary Care

David J. Kolko, Ph.D., ABPP, John V. Campo, M.D., Kelly Kelleher, M.D., MPH, and Yu Cheng, Ph.D.



To determine the effectiveness of an on-site modular intervention in improving access to mental health services and outcomes for children with behavioral problems in primary care relative to enhanced usual care.


Boys and girls from six primary care offices in metropolitan Pittsburgh, PA.


One-hundred and sixty three clinically referred children who met a modest clinical cutoff (75th percentile) on the externalizing behavior scale of the Pediatric Symptom Checklist-17 were randomized to a protocol for on-site, nurse-administered intervention (PONI) or to enhanced usual care (EUC). PONI applied treatment modules from an evidence based specialty mental health treatment for children with disruptive behavior disorders that were adapted for delivery in the primary care setting; EUC offered diagnostic assessment, recommendations, and facilitated referral to a specialty mental health provider in the community.

Main Outcome Measures

Standardized rating scales, including the PSC-17, individualized target behavior ratings, treatment termination reports, and diagnostic interviews were collected.


PONI cases were significantly more likely to receive and complete mental health services, reported fewer service barriers and more consumer satisfaction, and showed greater, albeit modest, improvements on just a few clinical outcomes that included remission for categorical behavioral disorders at one-year follow-up. Both conditions also reported several significant improvements on several clinical outcomes over time.


A psychosocial intervention for behavior problems that was delivered by nurses in the primary care setting is feasible, improves access to mental health services, and has some clinical efficacy. Options for enhancing clinical outcome include the use of multifaceted collaborative care interventions in the pediatric practice.

Keywords: Disruptive Behavior Disorders, Behavior Problems, Clinical Referral, On-Site Mental Health Services

The pediatric primary care setting is a critical venue for the identification and management of mental health problems,1 including behavior problems (BP) such as oppositional and aggressive behaviors. BPs are relevant to primary care clinicians (PCCs)2, 3 given their prevalence in pediatric practice (10%–17%), associated physical4 and psychosocial impairments,5 and relationship to excessive health and emergency services use.6, 7 Access to specialty mental health services is limited for children with BP and other common mental disorders such as ADHD.8, 9 Consequently, PCCs are increasingly being called on to address such problems,10 but report unease, inadequate training, limited tools,11 time pressures,5, 11 and minimal access to specialty mental health support 9, 12 as serious barriers to care delivery.13 It is, thus, important to determine if on-site mental health service delivery in primary care is feasible and improves access to care and clinical outcomes for children with BP.

Some efforts to deliver mental health services in pediatric primary care have shown promise in randomized trials.14 A study of internet-based psychoeducation and another of on-site family intervention for children with BP both documented success, 8, 15 whereas a trial involving preschoolers with Oppositional Defiant Disorder (ODD) found no significant advantages to an evidence-based treatment delivered either on-site (nurse) or in the specialty mental health setting (psychologist) relative to bibliotherapy.16 Two studies have shown modest responses by depressed adolescents to internet-based and on-site collaborative care interventions.17, 18 Work with depressed adults in primary care has also demonstrated that the provision of on-site services improves treatment engagement and outcomes relative to usual care.19, 20 Our own experience suggests heightened family compliance and satisfaction with services delivered by an on-site nurse clinician within a collaborative mental health team in a rural pediatric primary care practice.21

Given these experiences and research reports,2123 we sought to evaluate an intervention model for children with BPs that located a trained nurse in the primary care office who applied a modular intervention adapted for brevity and use in primary care from a treatment of pediatric BP with demonstrated efficacy.2427 The goal was to offer a feasible, efficient, and cost-effective alternative to the usual practice of off-site specialty mental health referral.28 We conducted a randomized controlled trial that compared a protocol for an office-based nurse-administered intervention (PONI) to enhanced usual care (EUC) that consisted of a facilitated referral to an off-site mental health professional and the provision of information from the initial psychiatric assessment and brief treatment recommendations to the PCC. Relative to EUC, we hypothesized that PONI would offer superior: 1) access to and engagement with mental health services; 2) child clinical outcomes at post-treatment and long-term follow-up; and, 3) client satisfaction with services.



The study enrolled 163 children ages 6 to 11 years old based on parent concerns about the child’s BPs and a parent rating at or above the clinical cutoff (6) for the 75th percentile on the externalizing behavior problems scale of the Pediatric Symptom Checklist (PSC-17). The study was approved by the IRB of the Children’s Hospital of Pittsburgh and University of Pittsburgh Medical Center; participants completed informed consent/assent as appropriate. Children with emergent psychiatric or medical needs (e.g., psychosis, suicidality with a plan, unstable medical illness) or actively participating in specialty mental health treatment were excluded. Approximately two thirds of participants were male, 80% Caucasian, and 75% living at home with married adults (see Table 1 for sample characteristics). A DSM IV psychiatric disorder was identified in 94% of the sample, the most common being ODD and ADHD; 91% met diagnostic criteria for a disruptive behavior disorder. Baseline levels of externalizing problems on the PSC-17 were high (M = 8.6; SD = 2.2). Most of the sample showed evidence of functional impairment and scored above PSC-17 clinical cutoffs for attentional and internalizing problems. Participants were randomized to PONI or EUC after initial assessment, and were well balpopoanced on all demographic and diagnostic characteristics

Table 1
Demographic, Clinical, and Diagnostic Characteristics of the Sample at Study Intake.


Participants were recruited from six large urban and suburban pediatric primary care practices covering 44,642 pediatric lives (range: 3,438–16,233), of whom 22,282 were within the age range of the study. Approximately 13.5% of all patients received Medical Assistance. The practices employed 30 pediatricians, 19 nurse clinicians, five medical assistants, and 37 office/administrative staff, and were members of Children’s Community Pediatrics (CCP), a pediatric primary care practice network affiliated with the Children’s Hospital of Pittsburgh. All 30 pediatricians confirmed a willingness to participate and follow study protocol. The practices did not have prior access to on-site mental health services or formal arrangements with local mental health providers.

Study Personnel


Because nurse clinicians with mental health training are in relatively short supply in most geographic regions, two registered nurses with at least one year of medical-surgical experience but without prior mental health training were hired as study clinicians to increase generalizability and potential for dissemination. Nurses received four months of ‘on-the-job’ training and ongoing supervision in diagnostic assessment and clinical intervention that included observation, role playing, in vivo practice, and feedback on videotaped interviews and clinical interactions specific to PONI and EUC. Nurses were also trained to complete a study database, interact with all office staff and PCCs, and participate in study case review and office practice meetings. Supervision by a Masters’-level clinician with experience administering the protocol was supplemented by access to a part-time child and adolescent psychiatrist for diagnostic/medical and medication questions. Each nurse was assigned to 3 practices, where they assisted with screening and initial assessments for all study participants prior to randomization and then completed treatment condition-specific tasks.

Research Assessors (RA)

Two bachelors-level research assessment staff, blind to child assignment, administered research interviews and assessments, with one RA administering rating scales and the other conducting the diagnostic interview. RAs received extensive training in using all measures and assessing safety risks and participated in weekly supervision and a diagnostic case review with the study psychiatrist. Recruitment and Assessment.

Potential subjects with BPs were identified by direct PCC referral, parental requests, or parental response to office screening or posted flyers in practice waiting areas. The nurse assigned to the practice conducted a screening interview (on-site or by phone) with the parent that established the child’s eligibility for age (5–12 yrs.), referral concern (behavior problems), and clinical severity (≥ 75th percentile) on the PSC-17 externalizing behavior scale. A study RA was intermittently available in practice waiting areas to conduct the screening interview if the nurse was not available. PCC endorsement also was necessary for eligible children to participate. If the child was eligible and the family willing, parent and child were scheduled for a research assessment conducted by two RAs who administered standardized study assessments. Upon completion of the research assessment, enrolled and eligible subjects were randomized to PONI or EUC.

Figure 1 depicts the overall flow of subjects across all study phases. Parents of 2,301 children expressed a general interest in the study, with 578 children being eligible. A total of 236 cases were able to be contacted and agreed to be scheduled for a formal research assessment, with 178 completing the assessment and fully meeting entry criteria. Of these 178 cases, 163 were enrolled in the study and randomized to PONI or EUC, of whom 173 (88%) completed the 12-month follow-up. Enrollment rates were evenly distributed across offices (M = 14%; 10–18%). Sources of initial contact with enrolled subjects included parental inquiry generated by office-based screening (n=90, 66%), flyers/advertisements (17), unsolicited parent request (6), and direct PCC referral (50).

Figure 1
Study Screening, Enrollment, and Participation Summary

Assessment Procedures

Research assessments were conducted with children and parents every 6 months (pre-treatment, post-treatment, and at 6- and 12-month follow-ups). Assessments lasted approximately 2.00 hours for children and 2.75 hours for caregivers. Each informant was paid in increasing amounts at each consecutive research assessment visit ($20, $30, $40, $50). All rating scales and diagnostic interviews were collected at the four research assessments. Pediatrician ratings were collected at the conclusion of the study.

Observations and Measurements

Clinical Problems, Impairment, and Adaptive Behaviors

Parent ratings on the 17-item PSC-17)2,3 yielded scores on three factors (externalizing behavior, attentional, and internalizing problems), each with a clinical cutoff.29 Parent and teacher reports on the Strength and Difficulties Questionnaire (SDQ)30 gave a comprehensive rating of the child’s clinical and peer problems. Child-reported anxiety symptoms were surveyed using the psychometrically sound 5-item short-form of the Scale for Anxiety and Related Emotional Disorders (SCARED).31 the short-form has good psychometric properties and a cutoff (3) for young clinical samples. Child-reported depressive symptoms in the previous 2 weeks were assessed using the 32-item (0–2 pt. scale) Mood and Feeling Questionnaire(MFQ).27, 32 The 13-item parent completed Columbia Impairment Scale (CIS) provided a global measure of child impairment in 4 areas of functioning (i.e., family, peer, work, school);33 the clinical cutoff is 15. Children completed the 45-item Child Health and Illness Profile CHIP34 to provide an assessment of overall adjustment across five primary domains and health-related behaviors. The domains include satisfaction (with health and self), comfort (physical and emotional), and resilience (physical activity, social problem solving).

Psychiatric Disorder

The Schedule for Affective Disorders and Schizophrenia for School-Aged Children for DSM-IV--Present and Lifetime (Kiddie-SADS-PL)35 was administered by trained diagnostic interviewers to detect current Axis I child and adolescent psychiatric disorders (e.g., ADHD, ODD, Major Depression). For each symptom which met the threshold criterion (rating of 3), several follow probes were asked (i.e., age of onset, age of offset, symptom frequency, problem settings, impairment level). In the case of ADHD, the diagnosis was rendered only if threshold symptoms were found at both home and school. Parents and children were interviewed separately and then jointly addressed any discrepancies. Inter-rater reliabilities for a subset of 36 cases were high (k’s = 0.75 – 1.00), which parallel our prior results with ODD and CD children.24

Individualized Treatment Goals

Parents were interviewed to identify treatment goals for up to four individualized child problems. At pre-treatment, each problem on the Individualized Goal Achievement Rating (IGAR) form 27 was defined. Specific behavioral anchors were obtained to define the level of improvement on a 5-pt scale (e.g., 1 = pre-treatment severity, 3 = expected level of improvement; 5 = exceeded expected level of improvement). All goals at pre-treatment were rated a “1”, but any 1–5 rating could be used subsequently to capture level of improvement per goal. Similar methods were used in our other behavioral outcome studies.27

Consumer Satisfaction and Barriers to Treatment. Parents completed the 8-item Client Satisfaction Questionnaire (CSQ-8)36 at post-treatment to capture perceptions of treatment acceptability and helpfulness, as well as the Barriers to Treatment Participation Scale (BTPS)37 to evaluate the impact of treatment obstacles or concerns in four domains (e.g., therapist relationship, competing activities) on a 1–5 point Likert scale.

Treatment Termination Summary (TSR)

The TSR was designed to document parameters of the child’s treatment. It was completed in PONI by all study nurses (n = 80) and in EUC by either an outside specialist (n = 26) or the child’s PCC (n = 54). The parameters included the receipt of the assigned treatment, key participants, type, dose, and duration of treatment, completer status at termination (i.e., case terminated upon reaching treatment goals or at a time mutually agreed upon with provider), case disposition, and any discharge recommendations. There was good agreement (kappa = 0.82) between the TSR medication status item and parental reports on a medication use item that was administered to parents at posttreatment (n = 114).

Pediatrician Perspectives on Mental Health Service Delivery

Pediatricians rated a few questions about their beliefs and opinions regarding the availability and relevance of on-site mental health services on Likert scales (1 = not at all; 5 = very much).

Treatment Conditions

Due to the small number of practices (6), randomization was by subject. Enrolled children meeting inclusion criteria were randomly assigned to PONI or EUC on a 1:1 basis. To ensure group comparability, Efron’s biased coin toss38 was used to balance the groups on gender, age (6–8 vs. 9–11), and a prior history of mental health treatment. Nurses reviewed assessment results and diagnostic impressions with the parents in both conditions, and gave a written summary of the initial assessment and recommendations to the PCC which was placed in the chart. Nurses attended practice meetings and adapted study procedures in accord with the patterns of each practice while preserving study integrity.

Protocol for On-Site, Nurse-Administered Behavioral Intervention (PONI)

The PONI condition was delivered in the practice by study nurses who coordinated the family’s care with the PCC. Families were informed that treatment would involve approximately 6 primary sessions (1.5 hrs each) for training in cognitive-behavioral treatment (CBT) skills and, as needed, 2–4 booster sessions to address emergent issues or promote maintenance on a key target. Thus, treatment was designed to be delivered in about 10 visits in 3–6 month. PONI cases were referred for outside mental health services only after study treatment had ended. The protocol incorporated “core” components in the treatment literature for BPs39 and whose content was found to be efficacious in our prior outcome studies for BPs,24, 25 child firesetting,40 and family aggression. 26, 42

Protocol content was developed in seven intervention modules: 42, 24 1) Child CBT. Children received training in the use of anger-control, self-instructions, or social skills supplemented with role-plays and home practice to promote self-management and prosocial behavior; 2) Medication for ADHD. Parents reviewed ADHD symptom ratings and the study psychiatrist’s medication recommendations with the nurse, and, if indicated, were encouraged to discuss medication treatment with their PCCs. PCCs could ask nurses to collect follow-up ratings and participate in a medication review session; 3) Enhanced Parent Management Training (PMT). Parents were trained to use positive reinforcement, non-corporal discipline involving contingency management (e.g., withdrawal of privileges), and anger management skills, to encourage appropriate behavior management; 4) Development: Expectations and Peer Enrichment. Parents received information on child development and age-appropriate behavioral expectations, and were taught to promote child involvement in peer and recreational activities; 5) Family Psychoeducation and Skills Training (PAST). Joint sessions discussed information about family rules, decision-making, and alternatives to the use of coercion or physical force, and then taught skills to enhance positive interactions (e.g., communication and problem-solving); 6) School/Teacher Consultation. Nurses contacted teachers to discuss relevant management or educational concerns, identify skills for remediation, and provide individualized behavioral suggestions based on teacher ratings; and 7) Case and Crisis Management. Nurses helped families respond to challenges and crises, and advocated with local service agencies and professionals, as needed.

The PMT, CBT, and PAST modules were administered first to teach basic skills, followed next by the remaining modules, as needed, to address problems related to ADHD, school misbehavior, problematic peer/community relations, and/or family crises/conflicts. The extent to which each module was administered reflected the severity of the child’s identified behavior problems and the family’s skill repertoire. Services were terminated when the family sought to end care or met treatment goals.

Enhanced Usual Care (EUC)

Children in the EUC condition were referred to an off-site specialty mental health provider. Study nurses identified local mental health providers based on a list generated by the practice, and facilitated referral by making an initial call to the provider that confirmed the family’s appropriateness (e.g., relevant clinical expertise) and acceptability for services (e.g., insurance). Referral information was conveyed to the PCC and documented in the medical record to facilitate PCP involvement in services, including medication prescription.

Study Attrition

Three PONI cases dropped out of the study before the first session, and six additional cases each in PONI and EUC dropped out after the first session. Overall study attrition was modest for PONI (11%) and EUC (8%), which compared to our prior clinic-based efforts (15%)25 and other clinical trials (10–14%).26, 27, 42

Treatment Integrity

We sought to minimize allegiance or contamination effects and enhance the integrity of both conditions using several steps: a) nurses were trained in both intervention procedures using specified manuals and clinical guidelines; b) nurses participated in weekly study team meetings which included discussions of overall study integrity but were not included in discussions of the study’s hypotheses or results; c) treatment adherence in both conditions was monitored routinely and weekly feedback given upon supervisor review of session notes and tapes; and d) we examined possible contamination effects at the level of the PCC.

In PONI, integrity was documented by a supervisor who rated a randomly selected set of treatment sessions (30%) using a Treatment Integrity Rating Form (TIRF).26 TIRF ratings reflecting the percentage of correctly administered content in each module were collected for 23 of 24 skill tasks assessed and found to be high (Mean = 91%; range: .68 – 1.00). The supervisor also reviewed each nurse’s Services Provided Log (SPL), which was completed after each treatment contact to document services received,26, 43 including any adjunctive services (e.g., crisis and case management). In EUC, the supervisor reviewed audiotapes of selected (15%) feedback sessions and an abbreviated SPL to document the completion of key EUC tasks and the absence of PONI content using a similar task list, which also yielded high integrity (Mean = 94%; .87 – 1.00). EUC families were contacted by the supervisor two weeks before post-treatment assessment to get permission to request a TSR from outside providers.

Because of concerns about contamination, we computed Intra-class Correlation Coefficients (ICC) reflecting the correlation among cases in the two conditions from the same PCC on the six primary child behavioral outcomes. Very low ICCs (M = 0.06) suggest little contamination by condition.

Data Analysis

Enrolled cases were followed per the intent-to-treat model. Descriptive analyses conducted for primary outcomes at pre-treatment found them to be normally distributed. Alphas for study measures were typically acceptable (0.60 – 0.93); the alpha for the SDQ-P conduct problems scale was modest (0.51). Rates of completed assessments were high at post-assessment (93%), six-months follow-up (91%), and one-year follow-up (88%).

To determine group comparability, we compared the groups on pre-treatment background and clinical variables using χ2 tests and ANOVAs.44 As preliminary analyses found no significant differences between the outcomes of the two nurses in each condition, the acute treatment outcomes of PONI and EUC were compared using χ2 tests and pre-post repeated measures ANOVAs. Chi-square tests examined the proportion of cases diagnosed with a psychiatric disorder at each assessment, and McNemar tests were used to examine changes over time in these proportions within conditions. For the subset of cases with ADHD at intake (n = 96), we also conducted 2-way repeated measures ANOVAs to evaluate the acute effects of medication (treatment condition × study medication) on all 11 child outcomes and Chi-square tests based on the rates of ADHD or ODD at posttreatment.

Repeated measures mixed effects models were conducted as primary analyses to evaluate the effectiveness of PONI (vs. EUC) on child outcomes through one-year follow-up. Descriptive analyses suggested that neither a linear nor quadratic curve model adequately described the change in outcomes over time. For most continuous outcomes, there was improvement from pre- to post-treatment, with change slowing during follow-up. Therefore, we adopted a repeated measures mixed-effect model with terms including treatment assignment, time as a factor, and the interactions of time and treatment assignment. We also included a random subject in the model allowing for different individual variability.


Treatment Involvement: Initiation, Participation, and Termination Status

On the TSR, 80 study nurses in PONI, and 24 specialists and eight PCPs in EUC, reported that they delivered services to the case. As shown in Table 2, children randomized to PONI were significantly more likely to initiate (access) their assigned treatment condition, receive more direct treatment and adjunctive services and a longer duration of treatment, and have sibling participation than those in EUC. PONI tended to have a greater number of sessions and a higher proportion of cases receiving psychosocial treatment only, although approximately three-quarters of all cases received psychosocial treatment only. Medication was prescribed for ADHD in 17 PONI children and 10 EUC children. PCPs prescribed the medication for all 17 PONI cases and 6 of the 10 EUC cases.

Table 2
Characteristics of the Services Provided in the Two Treatment Conditions (N = 163)

At termination, PONI children were more likely to complete their agreed-upon treatment regimens and to receive a case disposition indicating no further need for services or outside referral than EUC children. Across both conditions, non-completers and completers were comparable on all intake demographic and clinical variables.

Consumer Satisfaction and Barriers to Treatment

Caregivers of PONI (vs. EUC) cases reported significantly greater satisfaction (CSQ-8) and fewer obstacles to treatment on three of the four BTPS factors. Many parents in PONI noted the advantage of on-site services (e.g., acceptable, close to home).

Outcomes for Clinical Rating Scales

Acute Effects

Table 3 presents the results of the analyses comparing PONI and EUC on ratings of the child outcomes collected at pre- and post-treatment. Relative to EUC, PONI was associated with significantly greater, albeit modest, improvement on the IGA R severity rating of the child’s primary behavior problem and the overall CHIP adjustment score. Analyses revealed small to moderate improvements across time on all other ratings of behavioral and emotional problems for both conditions. EUC was not superior to PONI on any outcome measure.

Table 3
Child Clinical Outcomes at Pre-Assessment and Post-Assessment for the Two Treatment Conditions (N = 163)

We also examined the role of medication treatment for children with ADHD in the two conditions. In this subgroup, ANOVAs revealed no significant main effects of medication on the 6 primary behavioral and 5 secondary child outcomes (p’s < .78 - .08).

Follow-up Effects

Treatment effects across all time points were studied using repeated measures mixed effects models. Both treatments were associated with clinical improvements at follow-up on all 11 outcomes (p’s < .02 - .0001), with PONI showing superiority to EUC on the overall CHIP average score (F (4,552) = 2.61, p < .03) and tending to show greater reduction in teacher’s SDQ total scores (p < .06). To evaluate the maintenance of treatment gains, we compared the scores at post-treatment and one-year follow-up. Gains were generally sustained for most outcomes, with continued improvement documented on the IGAR average (t = 7.76, p < .0001), CIS (t = 3.33, p < .0001), MFQ (t = 5.97, p < 0.0001), SCARED-abbreviated (t = 2.7, p < .01), and teacher SDQ total scores (t = 2.29, p < .02).

Outcomes for Psychiatric Disorders

Acute Effects

As shown in Table 4, the proportion of PONI and EUC cases diagnosed with ADHD, ODD or CD, any mood/anxiety disorder, and any Axis I disorder were comparable at pre- and post-treatment. Figure 2 depicts the remission rates for ADHD and either ODD or CD at post-treatment. One-third of the cases in PONI and EUC were in remission for ADHD, and 53% and 41% had remitted from either ODD or CD, respectively. Based on McNemar tests, both conditions showed significant reductions in the rates of ADHD and ODD or CD at posttreatment (p’s < .001). ANOVAs comparing the mean number of diagnoses for PONI and EUC at pre-treatment (2.1, 2.0) and post-treatment (1.5, 1.4) revealed a significant effect over time (F = 48.36, p < .001), but no significant group x time interaction (F = 0.01, p < .89).

Figure 2
Remission Rates for Primary Disorders at Each Timepoint for the PONI and EUC Conditions.
Table 4
Psychiatric Diagnoses and Group Comparisons at each Timepoint (N = 163)

Follow-up Effects

As shown in Table 4, both conditions had comparable rates for all of the same disorders at one-year follow-up. As shown in Figure 2, the remission rates for ODD or CD in both conditions were even higher by follow-up (p’s < .001), but the remission rate for ADHD was significant for PONI (p < .001), and not EUC (p < .18). Overall, slightly less than one-half of all cases had met criteria for remission from ADHD and over one-half from ODD or CD.

Clinical Significance of Key Outcomes

As shown in Table 5, both conditions showed comparable proportions of cases whose scores on three key outcomes fell below the clinical cutoff at each assessment, with reductions found across timepoints. PONI demonstrated superiority to EUC in reducing total clinical problems on the parent SDQ at posttreatment and, to a lesser extent, one-year follow-up. The reductions in both conditions were maintained at one-year follow-up, with more than one-half of all cases falling within the normal range.

Table 5
Cases Scoring Below the Clinical Cutoff on Key Child Outcomes in the Two Treatment Conditions

Pediatrician Perspectives on Delivery of Mental Health Services

Pediatricians believed highly in the importance of making mental health services available in the primary care office (4.7/5.0), found few obstacles to providing such services (4.0/5.0), and expressed an interest in collaborating with an on-site clinician (5.0/5.0). PCCs expressed considerable enthusiasm with the delivery and expansion of the PONI condition.


This study evaluates the initial findings from a treatment trial targeting BPs in 163 clinically referred children (6–11 years) randomized to a protocol for on-site, nurse-administered intervention in primary care (PONI) or enhanced usual care (EUC) characterized by on-site diagnostic assessment and facilitated referral to a local mental health provider. EUC augmented traditional primary care management by reviewing results of a comprehensive diagnostic assessment with the patient, family, and managing PCC, and then aiding in linkage efforts. PONI (vs. EUC) cases were nevertheless more likely to access, complete, and be satisfied with the mental health services received, and reported greater improvements on a few clinical measures. But, both conditions experienced clinical benefits on several measures that were maintained through final follow-up. The relevance and scientific integrity of the study was enhanced by including children with disruptive disorders of concern to parents and PCCs, standardized assessment measures across multiple informants, and an evaluation of treatment integrity.

This study’s most encouraging finding is that PONI was associated with higher rates of treatment initiation and completion, a larger dose and duration of intervention, greater consumer satisfaction, and fewer barriers to treatment involvement compared to EUC. The rates of service initiation and follow-up achieved contrast with the low rates of specialty care referral (between 20 and 30%),3 service use (about a third), and follow-up visits (median = 2)12 reported in pediatric settings.11 Though less dramatic, PONI produced significantly better clinical outcomes than EUC in a few domains which included modest improvements in individualized behavioral treatment targets and overall child health/adaptive behavior. Both conditions showed reductions in externalizing, attentional, and internalizing problems and gains in social behavior and peer relations; one-third of children with ADHD and one-half with ODD or CD were in remission at post-treatment.

Follow-up analyses documented the maintenance of improvements in both conditions,8 including reductions in conduct and externalizing problems, attention problems, depression symptoms, functional impairment, and child health/adaptive behaviors. Indeed, one-half of the children diagnosed originally with ADHD or either ODD or CD met criteria for diagnostic remission by one-year follow-up. However, only PONI showed a significant decrease in cases showing clinically significant levels of problems at school and increase in remission from ADHD at one-year follow-up. These diagnostic outcomes are important to evaluate, but few treatment trials report them,

The findings from this clinical trial provide further support for the delivery of mental health services in pediatric primary care. The feasibility of on-site care by a pediatric nurse (PONI) was supported by high rates of treatment initiation and completion, especially a nearly 2-½ fold increase in service use over EUC, and its acceptability was demonstrated by reports of fewer barriers to treatment participation and higher consumer satisfaction. Further, PONI was at least as effective as EUC and was more effective on a few key outcomes at post-treatment and one-year follow-up, even though their clinical significance was small to modest. Although a cost analysis is beyond the scope of this report, it is worth mentioning that on-site services are now being delivered by and reimbursed for licensed clinicians in the same practices. In highlighting some advantages to the provision of on-site mental health interventions in pediatric primary care, these findings also support efforts to train nurses who may not have prior mental health experience as clinicians to deliver mental health services in pediatric primary care.8, 18

PONI’s modest outcomes are consistent with those from early trials of collaborative interventions for depressed adults in primary care,45, 46 and other pediatric trials,8, 18 highlighting the need for further evaluation and refinement of integrated mental health services for children with behavior problems. In particular, PONI might have been more effective had it included more active medication management, as only some children with ADHD received stimulant medication,47 or if intervention had continued beyond 6 months, as was often the case for EUC. Contingent upon PCP willingness and preparation, application of the chronic care model involving active care management, more active collaboration with PCCs, and greater attention to the treatment of comorbid conditions, might have been helpful.48 Accordingly, future efforts may benefit from a broader clinical agenda; formal PCC training in and decision support for medication use, especially for ADHD49, 50 or comorbid anxiety;51, 52,53, 54 and technological advances to enhance the efficiency with which on-site services are delivered and communicated.14

Study limitations deserve mention. The comparison EUC condition received care that was enriched by the availability of a comprehensive assessment, a diagnostic summary letter with recommendations, follow-up phone calls with referred families to confirm linkage, and extended services beyond the acute care period, making this a conservative test of PONI versus usual care. PONI provided only minimal attention to the clinical comorbidities of children with behavior problems, especially ADHD. We also recognize that the effects of treatment may vary by patient background, referral source, or other treatment variables.

In summary, a nurse-administered intervention applied in the pediatric primary care office was characterized by feasibility, integrity, acceptability, and modest clinical benefit, relative to a facilitated referral to an outside specialty provider. In extending findings from the few prior studies,8, 18 these promising results suggest the need for further study of innovative service models55 to address children’s psychosocial needs in primary healthcare settings, as is being done with ADHD.56 The improvements in access to care and clinical outcomes found for children in PONI provide a foundation upon which to develop and study more collaborative interventions whose quality may be improved by incorporating evidence-based programs that provide efficient screening, treatment, follow-up, and outside referral services. Although the application of an effective model of behavioral treatment in pediatric settings may not be a substitute for some specialty services, it may certainly help to define the foundation for effective mental health service provision in the primary care office.58


This study was supported, in part, by Grant MH63272 from the National Institute of Mental Health. We acknowledge the research and clinical staff of the Services for Kids In Primary Care (SKIP) program, the clinical and administrative staff affiliated with the participating pediatric practices from Children’s Community Pediatrics of Children’s Hospital of Pittsburgh, and the Advanced Center for Intervention Services Research (David Brent, MD, PI). Editorial assistance provided by James Shoemaker.

Contributor Information

David J. Kolko, Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine, Pittsburgh, PA.

John V. Campo, Nationwide Children’s Hospital and The Ohio State University, Columbus, OH.

Kelly Kelleher, Nationwide Children’s Hospital and The Ohio State University, Columbus, OH.

Yu Cheng, Western Psychiatric Institute and Clinic, University of Pittsburgh, Pittsburgh, PA.


1. Reigeisen H, Oliver KA, Menvielle E. Recognition and treatment of mental disorders in children. Pediatric Drugs. 2002;4:697–703. [PubMed]
2. Jellineck M, Murphy M, Little M, et al. Use of the pediatric symptom checklist to screen for psychosocial problems in pediatric primary care. Arch Pediatr Adolesc Med. 1999;153:254–60. [PubMed]
3. Gardner W, Murphy M, Childs G, et al. The PSC-17: A brief pediatric symptom checklist with psychosocial problem subscales. A report from PROS and ASPN. Ambulatory Child Health. 1999;5:225–36.
4. Garralda ME, Bowman F, Mandalia S. Children with psychiatric disorders who are frequent attenders to primary care. European Child and Adolescent Psychiatry. 1999;8:34–44. [PubMed]
5. Costello EJ. Child psychiatric disorders and their correlates: A primary pediatric sample. Journal of the American Academy of Child and Adolescent Psychiatry. 1989;28:851–55. [PubMed]
6. Zuckerman B, Moore KA, Glei D. Association between child behavior problems and frequent physician visits. Archive of Pediatric and Adolescent Medicine. 1996;150:146–53. [PubMed]
7. Lavigne JV, Arend R, Rosenbaum D, et al. Mental health service among young children receiving pediatric primary care. Journal of the American Academy of Child & Adolescent Psychiatry. 1998;37:1175–83. [PubMed]
8. Turner KMT, Sanders MR. Help when it’s needed first: A controlled evaluation of brief, preventive behavioral family intervention in a primary care setting. Behavior Therapy. 2006;37:131–42. [PubMed]
9. Kelleher K, Scholle SH, Feldman HM, et al. A fork in the road: Decision time for behavioral pediatrics. Journal of Developmental and Behavioral Pediatrics. 1999;20:181–86. [PubMed]
10. Kelleher K, Campo JV, Gardner W. Management of pediatric mental disorders in primary care: Where are we now and where are we going? Current Opinion in Pediatrics. 2006;18:649–53. [PubMed]
11. Gardner W, Kelleher K, Pajer K, et al. Primary care clinicians’ use of standardized psychiatric diagnoses. Child: Care, Health and Development. 2004;30:401–12. [PubMed]
12. Rushton JL, Forcier M, Schectman RM. Epidemiology of depressive symptoms in the National Longitudinal Study of Adolescent Health. Journal of the American Academy of Child & Adolescent Psychiatry. 2002;41:199–205. [PubMed]
13. Scholer SJ, Nix RL, Patterson B. Gaps in pediatricians’ advice to parents regarding early childhood aggression. Clinical Pediatrics. 2006;45:23–28. [PubMed]
14. Kelleher K, Stevens J. Evolution of child mental health services in primary care. Academic Pediatrics. 2009;9:7–14. [PMC free article] [PubMed]
15. Borowsky IW, Mozayeny SM, Stuenkel KN, et al. Effectiveness of a primary care-based intervention to reduce mental health problems in children. Pediatric Research. in press.
16. Lavigne JV, LeBailly SA, Gouze KR, et al. Predictor and Moderator Effects in the Treatment of Oppositional Defiant Disorder in Pediatric Primary Care. Journal of Pediatric Psychology. 2007;33:432–72. [PubMed]
17. Voorhees BW, Fogel J, Reinecke M, et al. Randomized clinical trial of an internet-based depression prevention program for adolescents (Project CATCH-IT) in Primary Care: 12-Week outcomes. Journal of Developmental and Behavioral Pediatrics. 2009;30:14. [PubMed]
18. Asarnow JR, Jaycox LH, Duan N, et al. Effectiveness of a quality improvement intervention for adolescent depression in primary care clinics. Journal of the American Medical Association. 2005;293:311–19. [PubMed]
19. Unutzer J, Katon W, Callahan C, et al. Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial. JAMA: Journal of the American Medical Association. 2002;288:2836–45. [PubMed]
20. Bartels S, Coakley E, Zubritsky C, et al. Improving access to geriatric mental health services: A randomized trial comparing treatment engagement with integrated versus enhanced referral care for depression, anxiety, and at-risk alcohol use. American Journal of Psychiatry. 2004;161:1455–62. [PubMed]
21. Campo JV, Shafer S, Strohm J, et al. Pediatric behavioral health in primary care: A collaborative approach. Journal of the American Psychiatric Nurses Association. 2005;11:276–82.
22. Silver EJ, Ireys HT, Bauman LJ, et al. Psychological outcomes of a support intervention in mothers of children with ongoing health conditions: The parent-to-parent network. Journal of Community Psychology. 1997;25:249–64.
23. Silverstein M, Mack C, Reavis N, et al. Effect of a clinic-based referral system to head start: A randomized control trial. Journal of the American Medical Association. 2004;292:968–71. [PubMed]
24. Kolko DJ, Dorn LD, Bukstein OG, et al. Community vs. clinic-based modular treatment of children with early-onset ODD or CD: A clinical trial with three-year follow-up. Journal of Abnormal Child Psychology. 2009;37:591–609. [PubMed]
25. Kolko DJ. Multimodal partial/day treatment of child antisocial behavior: Service description and multilevel program evaluation. Continuum. 1995;2:3–24.
26. Kolko DJ. Individual cognitive-behavioral treatment and family therapy for physically abused children and their offending parents: A comparison of clinical outcomes. Child Maltreatment. 1996;1:322–42.
27. Kolko DJ. Efficacy of cognitive-behavioral treatment and fire safety education for firesetting children: Initial and follow-up outcomes. Journal of Child Psychology and Psychiatry and Allied Disciplines. 2001:42. [PubMed]
28. Strosahl KD, Hayes SC, Bergan J, et al. Assessing the filed effectiveness of Acceptance and Commitment Therapy: An example of the manipulated training research method. Behavior Therapy. 1998;29:35–64.
29. Gardner W, Lucas A, Kolko DJ, et al. Comparison of the PSC-17 and alternative mental health screens in an at-risk primary care sample. Journal of the American Academy of Child & Adolescent Psychiatry. 2007;46:611–18. [PubMed]
30. Bourdon K, Goodman R, Rae D, et al. The Strengths and Difficulties Questionnaire: U.S. normative data and psychometric properties. Journal of the American Academy of Child & Adolescent Psychiatry. 2005;44:557–64. [PubMed]
31. Birmaher B, Khetarpal S, Brent D, et al. The screen for child anxiety related emotional disorders (SCARED): Scale construction and psychometric characteristics. Journal of the American Academy of Child & Adolescent Psychiatry. 1997;36:545–53. [PubMed]
32. Angold A, Costello EJ, Messer SC, et al. Development of a short questionnaire for use in epidemiological studies of depression in children and adolescents. International Journal of Methods in Psychiatric Research. 1995;5:237–49.
33. Bird H, Shaffer D, Fisher P, et al. The Columbia impairment scale (CIS): Pilot findings on a measure of global impairment for children and adolescents. International Journal of Methods in Psychiatric Research. 1993;3:167–76.
34. Rebok G, Riley A, Forrest C, et al. Elementary school-aged children’s reports of their health: A cognitive interviewing study. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care & Rehabilitation. 2001;10:59–70. [PubMed]
35. Kaufman, Birmaher B, Brent DA, et al. KIDDIE-SADS-present and lifetime version (K-SADS-PL) Instrument developed at Western Psychiatric Institute and Clinic; Pittsburgh, PA: 1996.
36. Attkisson CC, Greenfield TK, Sederer LI, Dickery B. The Client Satisfaction Questionnaire (CSQ) Scales and the Service Satisfaction Scale-30 (SSS-30) Outcomes Assessment in Clinical Practice. 1996:120–27.
37. Kazdin AE, Wassell G. Barriers to treatment participation and therapeutic change among children referred for conduct disorder. Journal of Clinical Child Psychology. 1999;28:160–72. [PubMed]
38. Efron B. Biostatistics casebook. Stanford, Il: Stanford University Press; 1980.
39. Kazdin AE. Child, parent, and family-based treatment of aggressive and antisocial child behavior. In: Hibbs ED, Jensen PS, editors. Psychosocial treatments for child and adolescent disorders: Empirically based strategies for clinical practice. 2. Washington, DC: American Psychological Association; 2005. pp. 445–76.
40. Kolko DJ, Watson S, Faust J. Fire safety/prevention skills training to reduce involvement with fire in young psychiatric inpatients: Preliminary findings. Behavior Therapy. 1991;22:269–84.
41. Kolko DJ, Swenson CC. Assessing and treating physically abused children and their families: A cognitive behavioral approach. Thousand Oaks, CA: Sage Publications; 2002.
42. Brent DA, Holder D, Kolko DJ, et al. A clinical psychotherapy trial for adolescent depression comparing cognitive, family, and supportive treatments. Archives of General Psychiatry. 1997;58:877–85. [PubMed]
43. Burns BJ, Farmer EMZ, Angold A, et al. A randomized trial of case management for youths with serious emotional disturbance. Journal of Clinical Child Psychology. 1996;25:476–86.
44. SPSS for Windows [computer program]. Version 16.0. Chicago, Il: 2007.
45. Smith GR, Monson RA, Ray DC. Psychiatric consultation in somatization disorder. New England Journal of Medicine. 1986;344:6. [PubMed]
46. Katon W, Von Korff M, Lin E, et al. A randomized trial of psychiatric consultation with distressed high-utilizers. General Hospital Psychiatry. 1992;14:8. [PubMed]
47. MTA Cooperative Group. A 14-month randomized clinical trial of treatment strategies for Attention Deficit/Hyperactivity Disorder. Archives of General Psychiatry. 1999;56:1073–86. [PubMed]
48. Weng H-C, Yuan B-C, Su Y-T, et al. Effectiveness of a nurse-led management programme for paediatric asthma in Taiwan. Journal of Paediatrics and Child Health. 2007;43:4. [PubMed]
49. Dunlop AL, Leroy Z, Trowbridge FL, et al. Improving Providers’ Assessment and Management of Childhood Overweight: Results of an Intervention. Ambulatory Pediatrics. 2007;7:4. [PubMed]
50. Gledhill J, Kramer T, Iliffe S, et al. Training general practitioners in the identification and management of adolescent depression with the consultation: A feasibility study. Journal of Adolescence. 2003;26:245–50. [PubMed]
51. Faust DS, Walker D, Sands M. Diagnosis and management of childhood bipolar disorder in the primary care setting. Clinical Pediatrics. 2006;45:801–08. [PubMed]
52. Rao G. Pediatric Obesity-Related Counseling in the Outpatient Setting. Ambulatory Pediatrics. 2005;5:2. [PubMed]
53. Weersing VR. Child Depression Consortium Meeting, sponsored by the Division of Pediatric Translational Research and Treatment Development, National Institute of Health and the Department of Psychiatry. University of Pittsburgh; Pittsburgh, Pennsylvania: 2007. Brief behavioral therapy for pediatric internalizing disorders: Results from a pilot randomized trial.
54. Weissman MM, Pilowsky DJ, Wickramaratne PJ, et al. Remissions in maternal depression and child psychopathology: A STAR*D-Child Report. Journal of the American Medical Association. 2006;295:1389–98. [PubMed]
55. Connor DF, McLaughlin TJ, Jeffers-Terry M, et al. Targeted child psychiatric services: A new model of pediatric primary clinician - child psychiatry collaborative care. Clinical Pediatrics. 2006;45:423–34. [PubMed]
56. Wolraich ML. Addressing behavior problems among school-aged children: Traditional and controversial approaches. Pediatrics in Review. 1997;18:266–70. [PubMed]
57. Jellinek MS. Commentary: Caring for the psychosocial needs of children: From advocacy to structural change. Ambulatory Pediatrics. 2008;8:8–10. [PubMed]