|Home | About | Journals | Submit | Contact Us | Français|
Do the GAPS comprehensive screening questionnaires increase detection or shorten time to diagnosis of mental health and substance abuse (MH/SA) disorders?
Analyses of an administrative database and a prior implementation study to compare change in MH/SA diagnoses pre-GAPS (May 1994 to April 1998) to post-GAPS (May 1999 to April 2003).
Rural health network in NY.
8,112 adolescents (13 to 15 years).
GAPS questionnaires routinely administered at annual visits starting May 1999.
Diagnostic codes reflecting MH/SA disorders for outpatient visits subsequent to an index annual visit were enumerated. Population based rates were derived using school enrollment data. Using time series, the rate of MH/SA diagnoses was compared pre- and post-GAPS. Using survival analysis, historical cohorts of adolescents ages 13–15 years who had an annual visit pre- or post-GAPS were constructed to compare the time to MH/SA diagnosis over two 4-year follow up periods. Using a merged database, a sensitivity, specificity, and proportional hazard analysis of the GAPS for MH/SA diagnosis was completed using 297 GAPS questionnaires.
The rate of MH/SA diagnosis did not change significantly pre- and post-GAPS (p=0.13). There was no difference in time to MH/SA diagnosis post-GAPS (7.0 months) and pre-GAPS (9.0 months, log rank p=0.30). A GAPS with > 3 items positive was associated with an increased risk of any MH/SA diagnosis (H.R. 1.97, p = 0.007).
Even though related to past or eventual MH/SA diagnosis, GAPS screening did not change the rate of or time to MH/SA diagnosis.
About half of all lifetime cases of mood, anxiety, impulse-control and substance-use disorders occur by age 14 years1. Thus the age of onset for lifetime mental health and substance abuse (MH/SA) disorders is concentrated in a relatively narrow age range in adolescence 2,3. Because the signs and symptoms of mental illness may be present two to four years before the onset of a full-blown disorder, early to middle adolescence is a window of opportunity for screening, detection and intervention4. Early identification of mental health problems has been linked to earlier treatment leading to better outcomes, particularly for adolescent depression 3,5 and substance abuse 6; however detection rates remain low7 . While it is widely assumed that requiring mental health screening will lead to greater recognition of under-diagnosed conditions and reduced unmet need for treatment, it is not currently known whether routine comprehensive screening increases the proportion of youth identified with MH/SA diagnoses or shortens the time to MH/SA diagnosis.
In 1993, the American Medical Association (AMA) produced the Guidelines for Adolescent Preventive Services (GAPS) as a clinical service tool for the provision of comprehensive adolescent health care8. Targeting the risk behaviors that contribute to the leading causes of adult morbidity and mortality, the GAPS model was intended to facilitate more effective risk identification, intervention and prevention by primary care providers (PCP). The GAPS model includes recommended services, PCP training materials, parent and youth pre-visit questionnaires and educational materials. The GAPS questionnaires (for adolescents ages 11 to 18 years) includes an inventory of adolescent health concerns, risk behaviors, mental health and substance abuse questions designed to detect problems that may require further assessment or referral for diagnosis and treatment 9. Although the GAPS mood related questions have been used to estimate the prevalence of depressive symptoms10, they are not diagnostic for depression. While there is no formal scoring of GAPS responses, the training manuals provide the PCP with guidance about how to respond to positive endorsements.
We had evaluated the implementation of GAPS in our clinic from 1998 to 2000 and found it to be a useful and feasible screening tool11. Use of the GAPS led to increased likelihood of documented discussion of risk behaviors during the visit, including psychosocial issues. The objective of this study was to determine whether introduction of the GAPS model and pre-visit screening questionnaires, and their long-term use in one rural pediatric clinic, led to increased detection and/or earlier MH/SA diagnosis. We hypothesized that use of the GAPS would increase the proportion of adolescents in primary care who were identified with MH/SA diagnosis compared to the pre-screening time period. Second, we hypothesized that the time to diagnose MH/SA would be shorter during the GAPS screening period compared to the pre-screening period.
Our Pediatric Clinic is located in the hub of a rural health network in central NYS. During the study period, the range of adolescents ages 13–15 seen per year was 440 to 568.
Three clinic staff members attended a training April 17–19, 1998 in Lisle, Illinois hosted by Arthur Elster, MD, who developed the GAPS model. Trained staff, in turn, trained the clinic PCP using the GAPS training materials. By May 1999, the Pediatric Clinic routinely administered the GAPS as a pre-visit screening questionnaire for all adolescent annual visits11.
The study design uses three analytic strategies to analyze two existing databases:
(1) interrupted time series of the rate of MH/SA diagnoses from 1994 to 2010 including 8,112 adolescents ages 13–15; (2) a historical cohort pre- and post-GAPS comparison of time to MH/SA diagnoses for 1,531 adolescents ages 13–15; (3) using a merged database, a sensitivity, specificity, and proportional hazard analysis of GAPS questionnaires completed by 297 adolescents ages 11–18 included in our prior implementation study11. Strategies 1 and 2 focused on the age range 13 to 15 years in order to include the peak time of age 14 years when half of lifetime cases of mood and substance-use disorders may occur1.
The databases analyzed included: (1) the Bassett Healthcare Network administrative database from 1994 to 2010 and (2) a GAPS Implementation Study database completed in 200211. The de-identified administrative database includes service location, dates of service, date of birth, gender, International Classification of Diseases, Clinical Modification (ICD-9-CM) diagnoses12, Current Procedural Terminology (CPT) codes13 and medical record numbers (MRN) for all visits in the network. The prior GAPS Implementation study database consisted of chart review data for a random sample (n=450) of adolescent annual visits from 1998 through 200011. It included MRN, dates of service, GAPS questionnaire responses and chart documentation. The post-GAPS years (1999 and 2000) each included 11% of all annual visits for ages 11–18 during which 297 GAPS questionnaires were completed11.
ICD-9-CM codes in the range of 291–314 were used to capture visit diagnoses that include common psychiatric and substance abuse disorders14. An MH/SA diagnosis was defined as any ICD-9-CM code in the 291–314 range that occurred on or after the index visit. If more than one ICD-9-CM code occurred in this range, the first one listed was used. Not Otherwise Specified (NOS) codes and Adjustment disorder (309.0–309.9) were included. Pervasive developmental disorders (299), developmental delays (315), psychiatric factors associated with physical diseases (316) and mental retardation (317–319) were excluded.
PCP may use alternate coding strategies for pediatric behavioral and mental disorders because of diagnostic uncertainty or sub-threshold symptoms, or to avoid denial of service15. These alternate codes include symptoms like “mood swings” or physical manifestations of mental problems rather than a specific MH/SA diagnosis. Wegner has described symptom related codes for behavioral and emotional problems that may be used by PCP to indicate a problem was identified but not diagnosed16. Appendix 1 includes the alternate codes used to capture PCP identification of a MH/SA problem in the time series and cohort analyses.
An interrupted time series was constructed to trend MH/SA diagnoses for adolescents ages 13–15 seen for an annual visit in the Pediatric Clinic pre- and post-GAPS introduction from 1994 to 2010. Well visit codes for all new (99384) and established patients (99394) were used to identify adolescent annual visits. Among youth who had an annual visit, MH/SA diagnoses were enumerated that were concurrent with the index visit, or occurred at subsequent outpatient visits at the Pediatric clinic or other sites in the network. A MH/SA diagnosis was defined as having at least one out of four possible codes defined as either an ICD-9-CM code in the 291–314 range, NOS or an alternate code used by PCP for a given outpatient visit. MH/SA visits were then restricted to five local zip codes that included most of youth seen in clinic. School enrollment data for grades 8, 9 and 10 in those five zip codes were used as denominators to derive population based rates. The rate of MH/SA diagnosis following an annual visit from 5/1/94 to 4/1/98 (pre-GAPS) was compared to 5/1/98 to 4/1/10 (post-GAPS) and tested for significance using the Chow test. The quality criteria for an interrupted time series study design were applied and met17.
Using the administrative database, cohorts of adolescents ages 13–15 who had an annual visit pre- or post-GAPS were constructed in order to compare the time to diagnosis over two follow-up periods. The index visit was defined as an annual visit identified using either CPT or ICD–9-CM code occurring either in pre-GAPS (May 1994 to April 1998) or post-GAPS (May 1999 to April 2003). In order to include youth with MH/SA diagnosis made at the index visit, the follow-up period began one day prior to the index visit. A four year follow-up time was chosen because the symptoms of mental illness may present two to four years before the onset of a full-blown disorder. Youth with pre-existing MH/SA diagnoses in the five months preceding the start times for these study periods were excluded. Due to the possible development of MH/SA diagnoses after five years, the pre-GAPS cohort was excluded from the post-GAPS cohort. The follow-up time was calculated as the time between the day before the index annual visit and the day of the last annual visit in the time period (right censored at last visit - no event), or time between the day before the first annual visit and the day of the MH/SA diagnosis (event). Using survival analysis, the median time to MH/SA diagnoses (291–314, NOS and/or alternate code) was compared pre- and post-GAPS for the two cohorts. Because the GAPS questionnaire includes specific screening questions for depressed mood and suicidality, diagnostic codes for mood disorders (depression 296.xx, depression NOS 311 and anxiety 300.xx) were analyzed separately.
The administrative database and the GAPS Implementation study database were merged using the MRN. Five variables were created to reflect varying levels at which the GAPS questionnaire could be considered positive. These included: 1) three or more risk questions endorsed positively; 2) one or more of the four mood questions endorsed positively; 3) a positive response to “felt sad, down, nothing to look forward to”; 4) a positive response to thoughts and/or plans for suicide; 5) any report of sexual, emotional or physical abuse. Looking back, sensitivity, specificity and predictive values of the GAPS were calculated for any prior MH/SA diagnosis (291–314 and NOS) occurring one year prior to the index visit when the GAPS was completed. Looking forward, proportional hazards analysis was used to estimate the risk of any MH/SA diagnosis (291–314 and NOS) or any mood disorder (depression 296.xx, depression NOS 311 and anxiety 300.xx) in the five years following the index visit. This method allows for incident MH/SA diagnoses that occur at different times after administration of the GAPS. Hazard ratios with 95% confidence intervals were estimated for each definition of a positive GAPS in order to estimate the risk of an MH/SA diagnosis with a positive GAPS as compared to a negative GAPS. Alternate codes were not used in these analyses in order to minimize diagnostic uncertainty.
This study was judged exempt from continuing review under section 46.101(b)(4) by the Bassett Hospital Institutional Review Committee.
The time series analysis included approximately 8,000 adolescents ages 13–15 from 1994 to 2010.The most common diagnoses were mood disorders, adjustment disorders and ADHD. There were only 18 SA diagnoses compared to 936 MH diagnoses; of the 18, nine were coded 305.1, i.e. non-dependent tobacco use disorder. The trend in MH/SA diagnoses pre-GAPS was significant compared to the trend post-GAPS (p=0.01, chow test), attributable to an increasing rate of MH/SA diagnoses occurring before GAPS was introduced (Figure 1). There was no significant trend in MH/SA diagnosis following its introduction and no significant change in rate of MH/SA diagnosis over the entire pre- and post-GAPS period (p=0.13).
Historical cohort analysis included 1,531adolescents ages 13–15 seen May 1994 through April 1998 (pre-GAPS) or May 1999 to April 2003 (post-GAPS) (Figure 2). In the five months preceding the start times for the pre- or post-GAPS periods, 66 adolescents (30 pre-GAPS, 36 post-GAPS) with pre-existing MH/SA diagnosis were excluded. The leading diagnoses for the 201 adolescents who acquired an MH/SA diagnosis during follow-up were mood disorders (27%), adjustment disorders (19%) and ADHD (15%). Event CPT codes included evaluation and management (32%), psychiatric (20%), labs and imaging (20%), annual and preventive (13%), vaccine-related (13%) and emergency department (ED) visits (2%). Survival analysis showed no difference in time to any MH/SA diagnosis post-GAPS (7.0 months) versus pre-GAPS (9.0 months, log rank p=0.30) (Figure 3). There was also no significant difference in time to any first diagnosis of mood disorder post–GAPS (12.2 months) versus pre-GAPS (11.0 months, log rank p=0.95) (Figure 4).
Sensitivity and specificity analysis included 297 adolescents ages 11–18 whose GAPS questionnaire responses were recorded in the implementation study11. The prevalence of MH/SA diagnosis in the 12 months prior to administration of the GAPS was 6.6%. The sensitivity for any past MH/SA diagnosis ranged from 86.7% if three or more GAPS items were positive to 7.1% if the abuse related item was positive (Table 1). The specificity ranged from 42.3% if three or more GAPS items were positive to 98.2% if the abuse related item was positive. The suicide related question had the highest positive predictive value (41.7%).
For mood disorders, the sensitivity and specificity of the GAPS was better, i.e. a sensitivity of 83% and a specificity of 82% for any past mood disorder if one or more of the mood-related items were positive (Table 2). However, there were only 12 cases of past mood disorder in this data set. Single item definitions for GAPS positive, i.e. depressed mood, suicide or abuse, had significantly lower sensitivity with a concomitant higher specificity.
For the 297 completed GAPS questionnaires, if three or more GAPS items were positive, the risk of any MH/SA diagnosis in five years was significant (H.R. = 1.97, 95% C.I. 1.20 – 3.23, p = 0.007) (Table 3). One or more positive mood related questions was associated with an increased risk of any MH/SA diagnosis (H.R. = 1.78, 95% C.I. 1.04 – 3.05, p = 0.03) (Table 3) and an increased risk of a mood related diagnosis (H.R. = 2.20, 95% C.I. 1.12 – 4.30, p = 0.02) (Table 4). Positive responses to depressed mood and abuse related single questions were not associated with an increased risk of MH/SA or mood disorder diagnosis (Tables 3 and and4).4). Positive response to the suicide related item was associated with both an increased risk of MH/SA diagnosis (H.R. = 3.29, 95% C.I. 1.51 – 7.18, p = 0.003) and mood disorder diagnosis (H.R. = 4.71, 95% C.I. 1.85 – 11.98, p = 0.001), however the confidence intervals were wide due to low frequency.
Using a variety of methods to ascertain the impact of GAPS screening on MH/SA diagnosis, we found no significant trend in overall MH/SA diagnosis during the study period. In this clinical sample, the use of the GAPS questionnaire was not associated with a significant change in trend or time to any MH/SA diagnosis or specific diagnosis of mood disorders. The increasing rate of MH/SA diagnoses pre-GAPS may have culminated in the introduction of the GAPS, but there was no significant trend in MH/SA diagnosis following its introduction. Sensitivity and specificity results suggest that the GAPS responses do relate to past or eventual MH diagnoses, particularly for mood disorders. Despite this, use of the GAPS does not appear to increase rate of or shorten time to MH/SA diagnosis pre- and post-GAPS use.
This study is unique in that it tests what we think the GAPS should be doing in terms of detection of MH/SA problems. By utilizing one pediatric practice, this study takes advantage of relative rural isolation and long term continuity of care for this source of care. The GAPS was administered consistently based on the implementation study. Clinic staff members received appropriate GAPS training so GAPS was implemented much like it would be elsewhere. However, our study results challenge the prevalent expectation that requiring mental health screening alone will lead to greater recognition of under-diagnosed conditions and reduce unmet need for treatment. Increased recognition may be a necessary, but not sufficient step, in shortening the time to diagnosis and therefore treatment. So the problem may not reside with the questionnaire itself or its administration, but rather with the subsequent steps needed to make a MH/SA diagnosis in this setting.
In the historical cohort analysis, the time to any MH/SA or to mood disorder diagnosis was similar pre- and post–GAPS. The fact this measure did not change post-GAPS suggests that the GAPS screening results may not be used effectively by either families or PCP’s. Alternatively, the MH/SA screening included in the GAPS may not be specific enough to trigger the subsequent steps needed to shorten time to diagnosis. If the PCP does not provide the MH/SA diagnosis, the mechanisms for acquiring a MH/SA diagnosis depend on subsequent steps of referral and gaining access to MH/SA services. Limited access to such services therefore could limit the rate of MH/SA diagnosis. The paucity of MH/SA professionals in this rural area may be one reason that MH/SA diagnoses did not increase despite the use of the GAPS in primary care.
The generalizability of this study is limited because the data are from one clinic in a rural health network in upstate NY. While this is a disadvantage, the relative paucity of other sources of care in the area may lead to a relative continuity of care, not necessarily by PCP, but within the rural health network. GAPS may have increased PCP awareness of sub-threshold emotional/behavioral problems for some youth. This may have actually led to “treatment”. However this study was only designed to examine MH/SA disorders that are at the diagnostic level.
Psychiatric hospitalization, and MH/SA diagnoses or treatment delivered outside our rural network, are not captured in this study’s databases. However, if diagnosed elsewhere, an adolescent’s MH/SA diagnosis is likely to be listed in subsequent primary care visits because follow-up usually takes place in primary care. This is particularly true if medication is required because in the study area, the PCP is often asked to continue prescribing psychiatric medications. Although serious MH/SA problems are much less common than depression and anxiety, adolescents with serious MH/SA problems may be less likely to come to clinic for annual visits and are more likely to present acutely. This study included MH/SA diagnoses made in our network’s ED’s as these visits are included in our administrative database, but does not include out-of-network ED visits.
This study suggests that the expected outcomes of MH/SA related screening in pediatric primary care settings need to be validated in order to ensure that screening is effective and is followed by the steps needed to accurately diagnose, appropriately refer and treat youth with MH/SA disorders.
This study was partially funded by the Center for Mental Health Services in Pediatric Primary Care (NIMH grant P20 MH086048).
296.99 (Mood swings, under other specified affective psychoses), 388.40 (Abnormal auditory perception, unspecified), 780.95 (Excessive crying of child, adolescent or adult), 783.0 (Anorexia), 783.1 (Abnormal weight gain), 783.21 (Abnormal loss of weight), 783.22 (Abnormal loss of weight and underweight), 783.9 (Growth/weight evaluation), 799.2 (Signs and symptoms involving emotional state excluding. anxiety and depression), 799.21 (Nervousness), 799.22 (Irritability), 799.23 (Impulsiveness), 799.24 (Emotional lability), 799.29 (Other signs and symptoms involving emotional state), 995.2 (Other and unspecified adverse effect of unspecified drug, medicinal and biologic substance), 995.20 (Unspecified adverse effect of unspecified drug, medicinal and biologic substance). V codes used during the study period that fit into the alternate coding group included: V40.0 (Problems w/learning), V40.1 (Problems w/communication), V40.2 (Other mental problems), V40.3 (Mental and behavioral problems; other behavioral problems), V60.0 (Lack of housing), V60.8 (Other specified housing or economic circumstances), V61.0 (Family disruption), V61.20 (Counseling for parent/child problem, unspecified), V62.3 (Educational circumstances), V62.5 (Other psychosocial circumstances; legal circumstances), V62.81 (Interpersonal problems, NEC), V62.82 (Bereavement, uncomplicated), V62.89 (Other psychological or physical stress, NEC, other), V65.42 (counseling on substance use and abuse), V65.49 (Other specified counseling), V65.5 (Person w/feared complaint in whom no dx was made), V71.02 (Observation for suspected mental condition; childhood or adolescent antisocial behavior), V71.09 (Other suspected mental condition).
Presented at Pediatric Academic Societies meeting, Boston April 28, 2012