Abstract
PURPOSE A risk prediction index, similar to those used for other disorders, such as cardiovascular disease, would facilitate depression prevention by identifying those who would benefit most from preventative measures in primary care settings.
METHODS The National Longitudinal Study of Adolescent Health enrolled a representative sample of US adolescents and included a baseline survey in 1995 and a 1-year follow-up survey in 1996 (n = 4,791). We used baseline risk factors (social and cognitive vulnerability and mood) to predict onset of a depressive episode at 1-year follow-up (eg, future risk of episode) and used boosted classification and regression trees to develop a prediction index, The Chicago Adolescent Depression Risk Assessment, suitable for a personal computer or hand-held device. True and false positives and negatives were determined based on concordance and discordance, respectively, between the prediction-category–based index and actual classification-category–based 1-year follow-up outcome. We evaluated the performance of the index for the entire sample and with several depressive episode outcomes using the standard Center for Epidemiologic Studies Depression (CES-D) scale cutoffs.
RESULTS The optimal prediction model (including depressed mood and social vulnerability) was a 20-item model with an area under the receiver operating characteristics curve of 0.80 (95% CI, 0.714–0.870), a sensitivity of 75%, and a specificity of 76.5%. For depressive episode, the positive predictive values in the highest risk group (level 4) was from 13.75% for a depressive episode to 63.57% for CES-D score of greater than 16 (mild to moderate depressed mood or above) at follow-up. Conversely, the negative predictive value of being in the lowest 2 levels (0 or 1) was 99.38% for a depressive episode and 89.19% for a CES-D score of greater than 16.
CONCLUSIONS Our model predicts a depressive episode and other depressive outcomes at 1-year follow-up. Positive and negative predictive values could enable primary care physicians and families to intervene on adolescents at highest risk.
INTRODUCTION
As many as 24% of adolescents will experience a depressive episode by the age of 24 years, with a median of onset age of 15 years.1 The onset of depression during this critical period may be particularly detrimental to the individual and society as a whole.2 Depressive illness, a recurrent life course disorder, is the second most costly disorder in developed countries with more than $100 billion in annual losses and expenditure in the United States alone.3 With full remission rates remaining less than 60% to 70%,4 reports in the United States and Europe and by the World Health Organization have called for the development of preventive interventions to reduce the burden of this disorder.5,6 Depressive disorders in adolescence are the largest single contributor to suicide risk during this period and the third leading cause of death in this age-group.7 Even with treatment, adolescents who have experienced one major depressive episode have higher risk of suicide and continued functional impairment.8,9
Preventive approaches in primary care hold the promise of potentially reducing both the risk of depressive disorder and risk of suicide in this age-group (Figure 1⇓) by intervening early in the sequence of depression pathogenesis. Risk for depression is derived substantially from demographic, behavioral, interpersonal, and cognitive vulnerability factors from within the adolescent’s social context. Both the American Psychological Association10,11 and the American Academy of Pediatrics have recommended the physician play a more active role in modifying social risk.12,13 Pediatricians and family physicians interact closely with adolescents and families within the community settings from which risk is derived and are increasingly be called upon to intervene with screening, referral, and treatment.10,14–16 A risk prediction index could enable primary care physicians to triage adolescents by risk level to an appropriate level of intervention (eg, face-to-face counseling for those at high risk,17 and perhaps Internet-based approaches (currently being evaluated by the author) for those at intermediate risk.)18–20
Prediction and early intervention model.
The purpose of this study was to develop a valid and practical risk prediction index (eg, future risk of episode), with a sensitivity and specificity of greater than 75%, using baseline vulnerability measures to predict new-onset depressive episodes at 1-year follow-up in US adolescents. We used the National Longitudinal Study of Adolescent Health data set because it is from the most recent longitudinal study of a national probability sample of US adolescents. The other major longitudinal studies either are either based on regional samples21,22 or do not include younger adolescents,1 and none is more recent.
METHODS
Sample and Survey Design
Mandated by the US Congress, The National Longitudinal Study of Adolescent Health (Ad Health) was a probability sample of US adolescents to provide information on the on the determinants and contexts of youth health and risk behaviors.23 Wave 1 included an in-home, face-to-face interview and an in-school survey in 1995 (76.6% enrollment rate; n = 6,504; public-use data set), and wave 2, a follow-up survey, was completed in 1996 (73.5% follow-up; n = 4,791). The surveys were organized by schools, and adolescents were selected by a stratified sampling method within schools. A particular focus in this study was obtaining an accurate understanding of the health of ethnic minorities. Consequently, ethnic minority youth were oversampled.24 Items used in the Ad Health surveys were developed based on multiple sources, and no complete scales were used.25 The present data analysis was approved by the University of Chicago Institutional Review Board and subsequently classified as exempt.
Outcome Variable
We sought to create an outcome that included all those individuals who had new cases of depressive disorder at 1-year follow-up. This outcome variable includes all youth who had 4 or more symptoms of depressive disorder at 1-year follow-up who were not already experiencing this level of symptoms at the baseline interview. The primary outcome variable of depressive episode was constructed using depression-specific items from the Center for Epidemiologic Studies Depression (CES-D) scale (20 items, score 0–60) at wave 2. Youth were considered to have a depressive episode if they met the following criteria: (1) reported experiencing at least 1 core symptom (depressed mood or anhedonia) “most all of the time for (5–7 days for the last week)” and (2) endorsed a similar level of severity for 4 or more other Diagnostic and Statistical Manual of Mental Disorders (Third Edition) criteria symptoms. We included within the depressive disorder outcome all those who may have major depressive disorder (5 or more symptoms) and those with severe minor depression (4 symptoms).26,27 We chose this cutoff because those with 4 or more symptoms are thought to warrant treatment intervention.28 To ensure that we were predicting only new-onset depressive episodes, we excluded those with a depressive episode at baseline (n = 100 who were present in both the wave-1 and wave-2 assessments).
Independent Variables
All 5,800 variables available at wave 1 were considered for inclusion in the model in a 2-step process. In the first step, 2 investigators (J.G. and B.W.V.) reviewed all available variables for analysis to determine which were either derived from standard instruments relating to depressive episode pathogenesis or were items that appeared to have face validity as potential predictors of future depressive episodes. These factors included sociodemographic characteristics (ethnicity, physical health, age), and interpersonal relations (social skills/problem solving deficits, lack of social support), interpersonal/family relations, and cognitive (pessimistic expectation, avoidant problem solving) and delinquency items.10,29,30 Current theoretical models of pathogenesis of depressive disorders formed the basis of this review (Figure 1⇑).10,30,31 We identified 119 variables by this method. Next, to select the variables from this list with fewest number of missing observations, we screened all variables to identify any with more than 10% of missing observations. We retained 52 variables, which enabled us to include 93.4% of the wave-2 sample in the analysis (Table 1⇓).
List of Variables Included in Analysis
Analysis
Boosted Classification and Regression Trees Method
We constructed a model using advanced predictive techniques (boosted regression and split-sample validation, training sample [60%], and a test sample [40%].) This method has shown satisfactory results predicting relatively rare outcomes involving complex interaction between independent variables.32,33 To evaluate the validity of the model, we report the sensitivity and specificity of the prediction model based on an optimal cutoff based on the receiver operator characteristics (ROC) curve. To develop the ROC curve, a risk score was calculated for each individual in the test sample based on the model derived from the training sample. The optimal point of the ROC curve achieves the greatest sensitivity and specificity (defined below). We report the ROC curve for the final model and the sensitivity and specificity of this prediction model at the optimal cutoff point in the test sample.
True and False Positives
True and false positives and negatives were determined based on concordance and discordance, respectively, between the index prediction category and the actual outcome classification category (based on follow-up outcome data). True positives and negatives were determined based on their prediction to be in the same category (eg, depressive episode at 1 year follow-up) and whether they actually were found to be in that category at follow-up. False positives and negatives were defined as cases predicted to be in the incorrect category at follow-up (eg, predicted to be unaffected at follow-up, but actually the individual had a depressive disorder at follow-up, false negative). Sensitivity, the ability to identify cases of depressive disorder, was defined by the following equation: Sensitivity = number of true positives/(true positives + false negatives). Specificity, the ability to classify “noncases” correctly, was defined by the equation: specificity = (true negatives/true negatives + false positives). We then converted these results to a 5-level prediction index for clinical use based on weighed probabilities derived from each individual predictor. We reported these probabilities for both the depressive episode outcome and those derived from standard cutoffs from CES-D scores representing clinically meaningful levels of severity.27 A more complete description of the methods can be found in Supplemental Appendix 1 and 2, available online at http://www.annfammed.org/cgi/content/full/6/6/503/DC1.
Positive and Negative Predictive Value
We selected the most parsimonious model with the maximum predictive power. Because many adolescents may be impaired at varying levels of depressed mood, we also report similar results (positive predictive values) for standard CES-D score cutoffs at greater than 16 (mild depressed mood or greater), greater than 23 (moderate depressed mood or greater), and greater than 28 and (severe depressed mood consistent with major depression).27 To help clinicians evaluate the clinical utility of this assessment tool, we calculated the positive predictive value for each level of risk predicted from baseline data for the test sample for depressive episode outcome and for standard CES-D score cutoffs. Similarly, we report the negative predictive value for the low-risk levels (0 and 1 combined). The positive predictive value represents the prevalence of cases at that level (eg, depressive disorder, CES-D score of greater than 16). The negative predictive value indicates the percentage of noncases of depressive disorder in those at the predefined level of risk. A more complete description of this method can be found in Supplemental Appendix 1 and 2. We used a Stata 9.0 plug-in to implement the boosted regression (StataCorp LP, College Station, Texas).
RESULTS
Study Sample and Comparison of Dropouts
The sample was a diverse, representative sample of US youth, 48% male (mean age 15.7 years), 57% white, 23% African American, 11% Hispanic, 1% American Indian, 3% Asian, and 5% multiracial. The mean family annual income was $47,700 (SD $3,097). With regard to parental education, 14.9% reported less than a high school degree, 53.4% a high school degree, and 31.6% graduated college or more. The study had an overall response rate of 76.8%, with a total sample of n = 6,504 (public-use data set, Sociometrics, Inc, Los Altos, California).24 Nearly three quarters (73.5%) of the wave-1 participants completed the wave-2 survey (n = 4,791).
Results from Boosting Regressions
When all 52 covariates were used, we found that a 6th-order interaction (6-way interactions among the independent variables) was the most optimal in terms of out-of-sample predictions. This model generated a pseudo-R2 of 0.159 in the test sample. Based on the predictions of this model, we ranked the covariates in terms of their degree of influence. Table 2⇓ provides a list of the top 20 covariates and their means and standard deviations for the training and test data sets. When only 20 covariates were used in the model, we found that a 3rd-order interaction was the most optimal in terms of out-of-sample predictions; it generated a pseudo-R2 of 0.122 in the test sample. The ROC curve generated using out-of-sample predictions from our 20-variable model with 3rd-order interactions is displayed in Figure 2⇓. The curve shows discrete changes in probability that result in a stairstep appearance. It produces an area under the curve of 0.80 (95% CI, 0.714–0.870), a sensitivity of 75%, and a specificity of 76.5% at a cutoff point of approximately 0.0177. Each of the top baseline 5 items strongly predicts the CES-D score in a linear regression model (Table 3⇓).
Top 20 Variables in Final Model in Order of Importance in Predicting a Depressive Episode at 1-Year Follow-up
Linear Regression Coefficients for the Top 5 Predictors of Depressive Episode at 1-Year Follow-up
Receiver operator characteristics curve using wave-1 independent variables to predict depressive episode at wave 2 in test data set.
CART=classification and regression trees.
Area under receiver operating characteristics curve = 0.80 (95% confidence interval, 0.71–0.87).
In test sample (N = 1,750; 40%).
Results From the Prediction Index
Finally, we developed the 5-level risk prediction scale (0 = very low risk, 4 = very high risk) based on the weighted average predictions of the boosted regressions (details given in the supplemental appendixes). These predictions generated an area under the curve of 0.78 (95% CI, 0.71–0.86) in the out-of-sample test data. Because of the skewed distribution of these weighted average predictions, we developed cutoffs based on the 10th (0.01932), 50th (0.02096), 90th (0.0272), and 95th (0.0309) percentiles of the predictions in the training data set to develop the 0 to 4 prediction scale. We evaluated other cutoffs and obtained similar results. We selected these cutoffs because they resulted in the optimal of the sample predictions. When we used the same cutoffs in our test data set, this scale performed remarkably well in discriminating among patients based on their observed likelihood of developing a depressive episode at a 1-year follow-up (Figure 3⇓). Using this index, the 1-year risk of developing a depressive episode is level 0 (very low risk, 0%), level 1 (low risk, 1%), level 2 (intermediate risk, 2%), level 3 (high risk, 5%), and level 4 (very high risk, 15%).
Predictive power of the 5-level prediction index in training and test sample.
Note: The y axis represents the predicted prevalence of depressive episodes at 1-year follow-up among subjects with a given level of score today.
We named this index the Chicago Adolescent Depression Risk Assessment (CADRA). Using the CADRA questionnaire and the Excel spreadsheet application suitable for a hand-held device or a personal computer (Microsoft Corp, Redmond, Washington), we calculated the baseline risk for each participant in the baseline survey and placed them in 5 levels of risk; the prediction index performed well for each depressive outcome examined (Table 4⇓). For depressive episodes, the positive predictive value for level 4 ranged from 15.0% for a depressive episode to 60.9% for a CES-D score of greater than 16 (mild to moderate depressed mood or above) at follow-up. Conversely, the negative predictive value of being in the lowest 2 levels (0 or 1) was 99.3% for a depressive episode and 89.4% for a CES-D score of greater than 16.
Positive and Negative Predictive Values for Depressive Episode and Standard CES-D Cutoff Scores
Supplementary Analyses to Evaluate Effects of Missing Data
Individuals missing from the analyses (primarily because they did not participate in the wave-2 survey) differed in some respects from those who were available for analysis. Nonresponders to the wave-2 survey were more likely to be African American, have lower income and lower parental education, and be slightly more depressed (CES-D score 11.2 vs 10.7, P >.011). The baseline mean values of the top 5 predictors of depressive episode at wave 2 differed by being slightly higher (more depressed) in those missing at wave 2 than those who were present and who were incorporated in these analyses.
DISCUSSION
This analysis shows the feasibility of developing risk prediction indexes for depressive episodes in adolescents. With a 20-item index (CADRA), we were able to achieve a greater than 75% sensitivity and specificity in predicting depressive episodes. CADRA also had excellent positive and negative predictive values for depressive episodes and varying levels of depressed mood (based on standard CES-D score cutoffs) at follow-up. Items found to be predictive included those related to mood, adverse events, and social vulnerability. These findings are consistent with current models that describe the pathogenesis of depression, suggesting that interactions between depressed mood, adverse events, and social vulnerability precipitate the onset of the disorder.10,30 Similarly, the high levels of interaction in the model are consistent with current theoretical models of causation.10,30
We are not aware of any current prediction indexes for depressive episodes intended for use in community or primary care settings for either adults or children. The sensitivity and specificity (92% and 62%) attained in this analysis are consistent with those of other prediction indices developed for adult-onset depression in much more homogenous samples with specific risk profiles (traumatic brain injury) and higher prevalence rates.34 In a study that evaluated the predicted value of multiple standard psychological distress questionnaires in predicting future depression in breast cancer patients, the best performance had a sensitivity and specificity of 76.9% and 78.8%, respectively, without a validation sample.35 Results of our study are also comparable with many other prediction rules and screening procedures currently used in primary care. For example the Framingham Risk Score has a sensitivity and specificity of 74.3% and an area under the ROC curve of 74.4,36 and the conventional Papanicolaou smear has a sensitivity and specificity of 76% and 68%, respectively.37 The Downtown fall risk index has a sensitivity ranging from 81% to 95% and specificity of 35% to 40%.38
The representative nature of the sample, the longitudinal design, the availability of multiple variables relating to established theoretical models of depression vulnerability and protection, and the boosted regression and split sample modeling are the key strengths of this study. There was a missing-data bias toward lower income and a greater likelihood of being ethnic minority and being slightly more depressed at baseline. The oversampling of ethnic minority groups, however, resulted in proportional overrepresentation in the sample. Although this data set is a decade old, and individuals missing from wave 2 tended to be more depressed than those who were analyzed, we believe fundamental relations between established vulnerability factors and depressive episodes are unlikely to change with time or differ meaningfully between those who were missing and those who were included in the analyses.29,39 Poor attachment to parents, low peer social acceptance, and baseline depressed mood or poor affect regulation have been identified as predictors of depressive disorders in multiple samples across historical periods and birth cohorts40,41 and in non-US samples42–44 and are the basis for current working models of depressive disorder onset.11,30,31 Although there has been a decline in substance abuse and other delinquent behaviors and a rise in the prevalence of adolescents living in nontraditional family structures, we do not believe these issues affected the risk factor of disorder relationships.45 This instrument will require validation and adaptation to hand-held device application (currently in progress in primary care population), and particular attention should be paid in this process to the experiences of ethnic minority youth.
As pediatricians and family physicians increasingly focus on optimizing development of prediction indices across all domains, including socioemotional health, such indices could play a critical role in preventing depressive episodes. Screening for mental disorders in youth is gaining increasing social acceptance, and the feasibility of screening for depressive episodes has been shown in primary care settings.14,15 Screening for depressive episodes will identify many mildly to moderately distressed youth for whom the family and physician would like to evaluate future risk of depression to appraise the need for early intervention (as is currently done with cardiovascular disease and breast cancer with paper indexes or hand-held devices). To illustrate, individuals at intermediate to highest risk for a depressive episode (level 4) in the upcoming year may be considered for face-to-face counseling for the adolescent and parents. Those at intermediate risk, levels 2 and 3, might be engaged in Internet-based learning programs based on cognitive behavioral interventions that have been evaluated in randomized trials in adults46 now being used in the United Kingdom47 and currently being tested in primary care here in United States in adolescents by the authors.18 Practical and reliable risk prediction for depression will increase awareness for the risk of depression and facilitate decisions regarding care for at-risk adolescents and their families.
Physicians and parents can access and use the risk index program at at http://catchit-public.bsd.uchicago.edu.”
Footnotes
-
Conflicts of interest: none reported
-
Financial support: Dr Van Voorhees is supported by a NARSAD Young Investigator Award, a Robert Wood Johnson Foundation Depression in Primary Care Value Grant, and a Career Development Award from the National Institute of Mental Health (NIMH K-08 MH 072918-01A2).
- Received for publication November 1, 2007.
- Revision received March 28, 2008.
- Accepted for publication April 2, 2008.
- © 2008 Annals of Family Medicine, Inc.