Article Text

PDF

The CBCL as a screen for psychiatric comorbidity in paediatric patients with ADHD
  1. J Biederman1,
  2. M C Monuteaux1,
  3. E Kendrick1,
  4. K L Klein1,
  5. S V Faraone2
  1. 1Pediatric Psychopharmacology Research Program, Massachusetts General Hospital, Boston, MA, USA
  2. 2Department of Psychiatry, SUNY Upstate Medical University, Syracuse, NY, USA
  1. Correspondence to:
    Dr J Biederman
    Massachusetts General Hospital, Pediatric Psychopharmacology Research Program, Yawkey Center for Patient Care-YAW-6A-6900, 32 Fruit Street, Boston, MA 02114, USA; jbiedermanpartners.org

Abstract

Aims: To examine the informativeness of the Child Behavior Checklist (CBCL) as a screening tool to identify comorbid and non-comorbid cases of attention deficit hyperactivity disorder (ADHD) in a paediatrically referred population. It was hypothesised that specific scales of the CBCL would help identify specific comorbidities within ADHD cases in the primary care setting.

Methods: The sample consisted of children and adolescents 6–17 years old of both genders with ADHD (n = 121). A receiver operating curve (ROC) approach was used to determine which CBCL scales best differentiated between ADHD cases with and without its comorbidities with conduct, anxiety, and mood disorders.

Results: ROC analysis showed that the CBCL Delinquent Behavior and Aggressive Behavior scales predicted the structured interview derived diagnoses of conduct and bipolar disorder, the Anxious/Depressed and Aggressive Behavior scales predicted major depression, and the Anxious/Depressed and Attention problems scales predicted anxiety disorders.

Conclusions: These results extend to a paediatrically referred population with previously reported findings in psychiatric samples documenting good convergence between structured interview diagnoses and syndrome congruent CBCL scales. These findings support the utility of the CBCL as a screening tool for the identification of psychiatric comorbidity in ADHD youth in the primary care setting.

Statistics from Altmetric.com

Converging evidence from clinical, school, and community samples documented that attention deficit hyperactivity disorder (ADHD) is heavily comorbid with disruptive (conduct and oppositional defiant disorder) (30–50%), mood (unipolar and bipolar) (15–75%), and anxiety disorders (20–30%).1–7 This high level of psychiatric comorbidity within ADHD has been clearly recognised in the guidelines of the American Academy of Child and Adolescent Psychiatry,8 the American Academy of Pediatrics,9 and European clinical guidelines.10

However, for non-mental health trained clinicians, distinguishing uncomplicated (simplex) from comorbid (or complex) cases of ADHD is a time consuming and difficult task. This issue is particularly relevant for paediatricians and routine care physicians that care for the majority of ADHD children.11,12 Thus, the identification of a useful screening tool to help primary care physicians discriminate comorbid from non-comorbid cases of ADHD would greatly aid the diagnosis and treatment of a large segment of children affected with ADHD. To be effective in this manner, this tool must be both easy to use and well validated. The empirically derived Child Behavior Checklist (CBCL)13,14 satisfies these criteria.

A large body of research shows the reliability and validity of the CBCL in clinical and non-clinical populations, both within the United States15–22 and elsewhere.23,24 The CBCL is a broad spectrum inventory that records, in standardised format, the behavioural and emotional problems and competencies of children aged 4–18, as reported by their parents or parent surrogates. It is scored on social competence and behaviour problem scales.14 The scales were originally constructed from analyses of parent ratings of 2300 clinically referred children and normed on 1300 non-referred children. Computer based programs can score the CBCL and generate T-scores on all subscales.

Moreover, because the CBCL is a paper and pencil instrument, it has the added advantage of minimising the primary care physician’s time, making it a cost effective clinical approach to identifying complicated ADHD cases. The paper and pencil approach of the CBCL stands in sharp contrast with the highly labour intensive demands of structured diagnostic interview methods.21,25,26

Our group previously examined the convergence of CBCL scales with specific structured interview based diagnoses in a sample of 114 youth.20,21 A significant congruence was found between the Attention Problems scale of the CBCL and structured interview derived diagnoses of ADHD, between the Anxious/Depressed CBCL scale and diagnoses of anxiety disorders, and between the CBCL Delinquent Behavior scale and diagnosis of conduct disorder. However, since this study was limited to psychiatrically referred cases, its informativeness as a screening tool to help identify comorbidity in the primary care setting cases remains unclear.

Efforts at facilitating the identification of comorbid from non-comorbid cases of ADHD in the primary care setting (that is, receiving routine medical care) have important clinical implications. Specifically, this would allow primary care practitioners to efficiently garner information regarding psychiatric comorbidity in their ADHD patients that in turn may affect the choice of treatment or referral for the afflicted child.

The purpose of the present study was to examine the informativeness of the CBCL as a screening tool to distinguish comorbid and non-comorbid cases of ADHD in a sample ascertained from a primary care population. We hypothesised that scales of the CBCL would help identify comorbidities within ADHD cases in the primary care setting.

METHODS

Detailed study methodology has been extensively described in previous publications.27–29 Briefly, the original sample consisted of children and adolescents 6–17 years old of both genders (n = 522), with and without ADHD ascertained from primary care (n = 290) and psychiatric (n = 232) clinics. For this study, we limited the analysis to ADHD probands ascertained from primary care settings (n = 141). Written informed consent was obtained for all subjects; children provided written assent to participate. The institutional review board approved this study.

Psychiatric assessments relied on the Kiddie SADS-E (Epidemiologic Version), a DSM-III-R-based structured interview.30 Diagnoses were based on independent interviews with the mothers and direct interviews of children, except for those younger than 12 years of age who were not directly interviewed. The data from direct and indirect interviews were combined by considering a diagnostic criterion positive if it was endorsed in either interview. The rates of illness reported here represent lifetime prevalence. All assessments were made by interviewers who were blind to the child’s diagnosis (ADHD or control) and ascertainment site (paediatric or psychiatric). Diagnoses were considered positive if, based on the interview results, DSM-III-R criteria31 were unequivocally met. All diagnostic uncertainties were resolved by a committee of board certified child and adult psychiatrists who were blind to the subject’s ascertainment group, ascertainment site, all data collected from other family members, and all non-diagnostic data (for example, cognitive functioning).

As suggested by others,32,33 the diagnosis of major depression was made only if the depressive episode was associated with marked impairment. Since the anxiety disorders comprise many syndromes with a wide range of severity, we report results for two or more anxiety disorders to index the presence of a meaningful anxiety syndrome.34

The mothers of each proband completed the 1991 version of the CBCL. The CBCL is an affordable pencil and paper test completed by the child’s caregiver, requiring no administration by a physician or rater. Use, scoring, and pricing information are easily accessible at: http://www.aseba.org/. In this study, no adjustments based on clinical concerns were made to the mothers’ ratings. The CBCL has nine behavioural problem subscales, and queries about the child’s behaviour in the past six months.13,14 The T-scores for each scale are calculated by a computer program. A T-score of 50 indicates average functioning in reference to other children of the same age and gender and every 10 points represents one standard deviation. In the interest of parsimony, we excluded the Somatic Complaints and Sexual Problems scales because preliminary analyses revealed their predictive properties to be far below the other scales.

Statistical analyses

First, we identified which CBCL scales were most effective as screening mechanisms for comorbid disorders common within ADHD: conduct disorder, major depression (with severe impairment), bipolar disorder, multiple (>2) anxiety disorder, and any comorbid disorder (any of the previous disorders versus none of these). We used a receiver operating characteristic (ROC) curve to assess the ability of each CBCL subscale to identify each comorbid condition. The ROC analysis uses each value across the entire range of the CBCL subscale T-scores as the cut-off for defining a case and compares this classification to the “true” diagnosis, as defined by the structured diagnostic interview. For this analysis, we collapsed the T-scores into categories (50, 51–59, 60–69, 70–79, ⩾80) in the interest of assessing practical, hands-on cut-offs that could be readily and easily applied to clinical settings. The ROC curve then plots the sensitivity versus the false positive rate (1-specificity) across the entire range of cut-offs. This analysis generates a statistic called the area under the curve (AUC), which is proportional to the overall ability of the scale across its range of cut-offs to correctly identify both cases and non-cases. For each ROC analysis, we compared subjects with the target disorder against all other subjects, even those with other disorders.

Second, we illustrated how specific cut-points of the scales performed as screening tests. For each comorbid disorder we chose the two subscales with the highest AUC values. With these two subscales we defined four cut-offs for defining a case: (1) both T-scores above 60; (2) the first T-score above 60 and the second T-score above 70; (3) the first T-score above 70 and the second T-score above 60; and (4) both T-scores above 70. We choose the cut-offs of 60 and 70 because they are one and two standard deviations above the mean, respectively, and are thus familiar to clinicians as landmarks along the scale of the T-distribution. The familiarity and intuitive value of these cut-offs should facilitate understanding and use. For each of these four cut-offs, we calculated the following statistics: (1) the positive predictive value (PPV): among the persons who received a positive screen, the proportion who in truth have the disorder; (2) the negative predictive value (NPV): among the persons who received a negative screen, the proportion who in truth do not have the disorder; (3) sensitivity: among the persons who have the disorder, the proportion who receive a positive screening test; and (4) specificity: among the persons who do not have the disorder, the proportion who receive a negative screening test. PPV and NPV assess the screening mechanism from a clinical perspective. For example, the PPV provides the probability that an ADHD child in a primary care setting who has a positive screen does indeed have the comorbid condition. Sensitivity and specificity are methodological measures of the screening test, useful in public health settings to make screening policy decisions. Together, they provide a comprehensive assessment of the CBCL scales as screens for psychopathology within ADHD.

RESULTS

A total of 121 paediatrically referred ADHD youth had CBCL data available, and are included in this analysis. The prevalence of comorbid disorders were as follows: conduct disorder (CD), 15% (n = 18); major depression (MD), 15% (n = 18); bipolar disorder (BPD), 7% (n = 8); and multiple anxiety disorder (ANX), 29% (n = 35). The prevalence of any of these comorbid disorders was 44% (n = 53). Hereafter, these comorbid cases are referred to as complex ADHD, while the non-comorbid cases (56%, n = 68) are referred to as simplex ADHD.

As shown in table 1, the complex ADHD group was significantly older than the simplex group (mean difference of 1.5 years). However, no significant differences between the two groups were found in SES, gender, or intactness of the family.

Table 1

 Demographic characteristics of paediatrically referred ADHD youth with and without psychiatric comorbidity

To identify which scales best discriminate specific comorbidities, we examined the area under the curve (AUC) using receiver operating characteristic (ROC) analysis as described above (see table 2). For CD, the Delinquent Behavior scale and the Aggressive Behavior scale yielded the greatest AUCs (0.81 and 0.80, respectively). In other words, there is an 80% chance that the Aggressive Behavior scale T-score of a randomly selected ADHD child with CD will be greater than the Aggressive Behavior scale T-score of a randomly selected ADHD child without CD. Similarly, BPD was also best predicted by the Delinquent Behavior scale and the Aggressive Behavior scale. The Anxious/Depressed scale and the Aggressive Behavior scale best predicted MD and any comorbid disorder. For anxiety disorders, the Anxious/Depressed scale and Thought Problems scale were found to have the highest AUCs.

Table 2

 The area under the curve and 95% confidence interval from ROC analysis using CBCL scales to predict psychiatric comorbidity in a paediatrically referred ADHD sample (n = 121)

In table 3, we display the performance of the identified scales to screen for each comorbid disorder. As the cut-off became greater (that is, a more stringent screening test), the PPV and specificity increased while the NPV and sensitivity decreased. For CD, the most efficient cut-off points were an Aggressive Behavior score >60 and a Delinquent Behavior score >70; these resulted in an PPV and NPV of 86% and 90%, respectively. In other words, of all ADHD children who received a positive screen, 86% would be diagnosed with CD by a structured interview. Likewise, of all ADHD children who received a negative screen, 90% would not have a structured interview derived diagnosis of CD. Thus, using these cut-offs on these scales would result in a very few ADHD children being misidentified and in very few true cases being missed. These analyses also showed that specificity was consistently high, ranging from 80% to 99% whereas sensitivity varied from 24% to 82%. Thus, these scales performed much better in ruling out a diagnosis of CD (that is, identifying non-CD children), but more variably in ruling in a diagnosis of CD (that is, identifying true CD cases). As shown in table 3, for BPD, MD, and ANX the PPVs and sensitivities were relatively low, but the NPVs and specificities were relatively high, indicating that the test is much better at ruling out these specific disorders than ruling them in.

Table 3

 Sensitivity, specificity, positive predictive values, and negative predictive values in the use of CBCL scales for the screening of psychiatric comorbidity in paediatrically referred ADHD youth

DISCUSSION

In a sample of paediatrically referred ADHD probands, ROC analysis indicated that the CBCL Delinquent Behavior and Aggressive Behavior scales best converged with the structured interview derived diagnoses of CD and BPD, the Anxious/Depressed and Aggressive Behavior scales corresponded with major depression, and the Anxious/Depressed and Attention Problem scales corresponded with anxiety disorders. These results extend to a paediatrically referred population previously reported findings in psychiatric samples documenting good convergence between structured interview diagnoses and syndrome congruent CBCL scales.

These results in paediatrically referred children are remarkably consistent with our previous work in a psychiatrically referred sample.21 Additionally, the present results agree with those presented by Steingard and colleagues20 who showed that CBCL scores in children with ADHD and associated comorbidity were significantly more impaired compared to those of ADHD children without comorbidity. Taken together, these findings support the utility of the CBCL to identify patterns of comorbidities within the context of ADHD.

Our new ROC analysis showed that the Delinquent Behavior and Aggressive Behavior scales were the best predictors of CD. Similar results were found by Kasius and colleagues19 who also reported good convergence with both the CBCL Aggressive Behavior and Delinquent Behavior scales with the diagnosis of CD. Conditional probability analysis indicated that these scales have very good discriminating utility to screen for CD.

The Delinquent Behavior and Aggressive Behavior scales of the CBCL were also the most informative in the prediction of BPD. These findings bear striking similarities to a previous study that examined the utility of the CBCL to identify preadolescent children with paediatric bipolar disorder25 in a psychiatrically referred population. That study also found significant increases in the Delinquent Behavior, Aggressive Behavior, and Anxious/Depressed scales as well as the Thought Problems scales compared with findings in children with ADHD, supporting the hypothesis that the CBCL was helpful in differentiating paediatric bipolar disorder from ADHD. Similar results were reported by Geller and colleagues35 and further expanded on by Hazell and colleagues36 and summarised by Mick and colleagues37 using a meta-analysis.

Our results also showed good convergence between the CBCL Anxious/Depressed scale and the presence of anxiety disorders. This finding is consistent with previous work in psychiatric samples.19,21

ROC and conditional probability analyses revealed that the CBCL performed reasonably well as a screening tool for comorbidity in ADHD children in the primary care setting. For example, the PPV and NPV of 65% and 77% for the CBCL Anxious/Depressed scale and Attention Problems scale were fair screeners of anxiety disorders, indicating that 65% of cases screened positive and 23% of those screened negative would be diagnosed with anxiety disorders. Likewise, conditional probability analysis showed that the Anxious/Depressed scale and Attention Problems scale were fair screeners for major depression in children in the primary care setting, with a PPV and NPV of 65% and 77% respectively.

The presence of at least one comorbid disorder (Any Comorbidity) was best predicted by the Aggressive Behavior scale and the Anxious/Depressed scale. Conditional probability analysis showed that these scales performed fairly well as a screen for at least one comorbidity when using a cut-off of >60 on the Aggressive Behavior scale and >70 on the Anxious/Depressed scale. These cut-off points yielded a PPV and NPV of 80% and 63% respectively indicating that 80% of cases screened positive for either CD, BPD, anxiety disorders, or major depression by the CBCL and 27% of those screened negatively would have one of these disorders when assessed by structured diagnostic interview. These results suggest that the CBCL is a viable option for screening ADHD children for psychiatric comorbidity in the primary care setting.

When evaluating the performance of the CBCL as a screening tool, it is important to consider the effect of prevalence on PPV and NPV. As shown in fig 1, given constant sensitivities and specificities of 25% and 95%, respectively, the PPV and NPV vary according to prevalence. That is, the PPV increases as prevalence increases, while the NPV decreases as prevalence increases. Thus, when considering the CBCL as a screening tool in a given clinical setting, the base rate of comorbid disorders in the service population should be taken into account.

Figure 1

 Effect of disease prevalence on positive and negative predictive values with sensitivity and specificity set to 25% and 95%, respectively.

The CBCL achieved very high specificity (>90%), but generally low sensitivity. Given these results, the utility of the CBCL as a screening device can only be defended if the benefit of detecting a minority of cases offsets the costs of the screen coupled with the costs of false positives. That is, false positive cases lead to unnecessary costs in psychiatric follow up evaluations, a shortcoming that needs to be weighed against the financial and human burden of failing to identify and treat a comorbid disorder that would have gone undetected if the screening programme were not undertaken. Given the low rate of false positives, the affordability of the CBCL, and the human and financial burden of psychiatric disorders in youth, this cost-benefit ratio is likely to be favourable. Also, it is possible that a psychiatric evaluation for a false positive screen may serendipitously detect another psychiatric disorder that was not screened for, which adds further benefit to the screening programme.

These findings should be viewed in light of some methodological limitations. Since our subjects were mostly Caucasian, our findings may not generalise to minorities. Also, since the sample consisted of children that were referred, our findings may not generalise to community samples. Additionally, the relatively small number of bipolar affected subjects (n = 8) may have limited our ability to evaluate the utility of the CBCL to screen for this disorder. For each disorder, we chose the two scales with the largest AUCs. Although more formal, statistical methods are available to compare ROC curves, we chose not to incorporate statistical inference into this selection process because of the large number of tests we would have to conduct and the inflated type I error rate that would follow. Also, we used the 1991 version of the CBCL. Clinicians would likely be using the 2001 version, and thus the scale may perform differently. However, the updates are minor and any changes in performance should also be minimal. Finally, the mother of each child completed both the CBCL and the structured diagnostic interview. The performance of the CBCL against another validation standard (that is, a teacher report or a physician rating) may yield different results.

What is already known on this topic

  • ADHD is highly comorbid with other psychiatric disorders

  • The clinical scales of the CBCL are correlated with psychiatric diagnoses in psychiatric samples, including ADHD samples

What this study adds

  • The CBCL is useful as a screen for comorbid disorders in a paediatrically referred ADHD sample

Despite these considerations, our results show good convergence between syndrome congruent scales of the CBCL with structured interview derived diagnoses of CD, BPD, MD, and anxiety disorders in paediatrically referred ADHD children. These results suggests that the CBCL may be a useful screening tool to help identify key comorbid disorders in the primary care setting, especially CD and at least one major comorbid disorder, and as such could be a useful diagnostic aid for primary care physicians.

REFERENCES

View Abstract

Footnotes

  • Dr Stephen V Faraone receives research support from the following sources: McNeil Consumer & Specialty Pharmaceuticals, Shire Laboratories, Eli Lilly & Company, the National Institute of Mental Health, The National Institute of Child Health and Development, and the National Institute of Neurological Diseases and Stroke. He is a speaker for the following speaker’s bureaus: Eli Lilly & Company, McNeil Consumer & Specialty Pharmaceuticals, and Shire Laboratories. He has had an advisory or consulting relationship with the following pharmaceutical companies: McNeil Consumer & Specialty Pharmaceuticals, Noven Pharmaceuticals, Shire Laboratories, and Eli Lilly & Company.

  • Funding: this work was supported, in part, by USPHS (NIMH) grant R01MH-41314 (JB)

  • Competing interests: Dr Joseph Biederman receives research support from the following sources: Shire Laboratories, Inc and Eli Lilly & Company, Pfizer Pharmaceutical, Cephalon Pharmaceutical, Janssen Pharaceutical, Neurosearch. Pharmaceuticals, Stanley Medical Institute, Lilly Foundation, Prechter Foundation, NIMH, NICHD, and NIDA. He is a speaker for the following speaker’s bureaus: Eli Lilly & Company, Pfizer Pharmaceutical, Novartis Pharmaceutical, Wyeth Ayerst, Shire Laboratories Inc., McNeil Pharmaceutical, and Cephalon Pharmaceutical. He is on the advisory board for the following pharmaceutical companies: Eli Lilly & Company, CellTech, Shire Laboratories Inc., Novartis Pharmaceutical, Noven Pharmaceutical, McNeil Pharmaceuticals, Janssen, Johnson & Johnson, Pfizer, and Cephalon Pharmaceuticals.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Atoms
    Howard Bauchner