Article Text

PDF

Meta-analysis of bacterial meningitis score validation studies
  1. Lise E Nigrovic1,
  2. Richard Malley1,2,
  3. Nathan Kuppermann3
  1. 1Division of Emergency Medicine, Department of Pediatrics, Children's Hospital Boston and Harvard Medical School, Boston, Massachusetts, USA
  2. 2Division of Infectious Diseases, Department of Pediatrics, Children's Hospital Boston and Harvard Medical School, Boston, Massachusetts, USA
  3. 3Departments of Emergency Medicine and Pediatrics, University of California, Davis School of Medicine,, Davis, California, USA
  1. Correspondence to Lise E Nigrovic, Division of Emergency Medicine, Department of Pediatrics, Children's Hospital Boston and Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA; lise.nigrovic{at}childrens. harvard.edu

Abstract

Objective The Bacterial Meningitis Score, a derived and validated clinical decision rule, identifies children with cerebrospinal fluid (CSF) pleocytosis who are at very low risk of bacterial meningitis. Low-risk features include the following: negative CSF Gram stain, CSF absolute neutrophil count (ANC) <1000 cells/μl, CSF protein <80 mg/dl, peripheral blood ANC <10 000 cells/μl and no seizure at or prior to initial presentation. The study objective of the present work was to calculate the performance of the Bacterial Meningitis Score by performing a meta-analysis of all published validation studies.

Methods A meta-analysis of all studies published between 2002 and 2012 was performed, evaluating the performance of the Bacterial Meningitis Score in children with CSF pleocytosis. Study quality was assessed using the assessment of diagnostic accuracy studies instrument and then the test performance of the prediction rule was calculated.

Results From 8 studies, 5312 patients were identified, of whom 4896 (92%) had sufficient clinical data to calculate the Bacterial Meningitis Score. Bacterial meningitis was diagnosed in 1242 children (23% of study patients). The combined sensitivity of the Bacterial Meningitis Score for bacterial meningitis was 99.3% (1224/1233; 95% CI 98.7% to 99.7%), specificity 62.1% (2274/3663; 95% CI 60.5% to 63.7%) negative predictive value 99.7% (2274/2283, 95% CI 99.3% to 99.9%), positive likelihood ratio 2.6 (95% CI 2.5 to 2.7) and negative likelihood ratio 0.01 (95% CI 0.01 to 0.02).

Conclusions The Bacterial Meningitis Score is a highly accurate clinical scoring system that could be used to assist clinical decision making for the management of children with CSF pleocytosis.

View Full Text

Statistics from Altmetric.com

Introduction

In regions of the world with high vaccination rates, the incidence of bacterial meningitis has declined substantially due to highly effective conjugate vaccines.1,,5 However, children with cerebrospinal fluid (CSF) pleocytosis are frequently hospitalised and given broad-spectrum antibiotics while awaiting the results of bacterial cultures, which may take 48 h to reliably exclude bacterial growth.6,,8

Children at very low risk for bacterial meningitis can be considered for outpatient management if they are otherwise well appearing and have adequate clinical follow-up. The Bacterial Meningitis Score clinical prediction rule was derived and internally validated from a retrospective cohort of 696 children with CSF pleocytosis hospitalised at a single institution.9 Although the Bacterial Meningitis Score performed with very high accuracy, clinical application was limited by the small study sample size, single-centre design with a highly referred population, lack of external validation, as well as ongoing changes in the epidemiology of bacterial meningitis related to the introduction of bacterial conjugate vaccines.5

What is already known on this subject

The Bacterial Meningitis Score, a previously derived and validated clinical prediction rule, identifies children with cerebrospinal fluid (CSF) pleocytosis who are at very low risk for bacterial meningitis.

What this study adds

The Bacterial Meningitis Score performed well in eight published validation studies and could be used to assist clinical decision making for children with CSF pleocytosis.

The ‘real-world’ performance of a clinical prediction rule is most accurately assessed by its application in a variety of clinical settings. We initially tested the Bacterial Meningitis Score in a large multicentre US study of children with CSF pleocytosis.10 The Bacterial Meningitis Score has also been evaluated in six additional studies by independent investigators.11,,16 In this study, we sought to measure the accuracy of the Bacterial Meningitis Score by aggregating the patients from the eight validation studies and to report the performance of the prediction rule in the combined population.

Methods

Study design

We performed a fixed-effects meta-analysis of the published Bacterial Meningitis Score validation studies.17 ,18 We searched the Medline and Embase electronic databases for eligible articles published between October 2002 and March 2012. We used the following search terms: Bacterial Meningitis Score, bacterial meningitis prediction and meningitis validation study. Additionally, we reviewed all publications that referenced the derivation study.9 One publication that is currently in press was identified by a published abstract and we subsequently communicated with the corresponding author. We reviewed potentially eligible studies to identify those that included children younger than 18 years of age as well as sufficient information to calculate the Bacterial Meningitis Score. We excluded patients used for the prediction model derivation conducted by the study investigators.9

Data collection

We reviewed eligible studies to determine study design, inclusion and exclusion criteria, patient population and case definitions. Because antibiotic pretreatment can render bacterial cultures falsely negative19 ,20 and also impact CSF profiles,21 we excluded studies in which patients had received antibiotics prior to lumbar puncture (LP) (defined as ‘antibiotic pretreatment’). We contacted the corresponding authors of the published studies to clarify study methods, as necessary. We excluded studies for which we could not verify study procedures. We assessed the quality of the included studies using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) instrument.22

Bacterial meningitis score performance

For each of the included studies, we determined the number of patients with bacterial and aseptic meningitis using study-specific case definitions. Children with none of the five high-risk Bacterial Meningitis Score predictors were classified at ‘very low risk’ for bacterial meningitis (table 1).9 ,10 Children with one or more high-risk predictor were classified as ‘not low risk’ even if other predictors were missing. Otherwise, children missing predictors included in the Bacterial Meningitis Score were excluded from the prediction rule validation.

Table 1

Bacterial Meningitis Score9

Statistical analysis

We abstracted the Bacterial Meningitis Score as calculated by the study investigators. We then calculated the performance of the dichotomised Bacterial Meningitis Score (‘very low risk’ vs ‘not very low risk’) in the aggregated patient population. Our primary outcome was the prediction rule sensitivity, which we calculated for each of the included studies. Our secondary outcomes were specificity, negative predictive value (NPV), positive predictive value (PPV) as well as positive and negative likelihood ratios. We used patients from case-control and cohort studies for calculation of sensitivity and specificity, but not for PPV and NPV. We used children from cohort studies for the calculation of sensitivity, specificity, NPV, PPV as well as likelihood ratios using standard techniques.23

We present the pooled effects as a point estimate with 95% CI using binomial methods with a Forest plot for the primary outcome measure. We calculated the Q statistic (χ2-distributed with one less than the number of included studies degrees of freedom) to assess the heterogeneity between studies as well as the I2 statistic to assess for consistency of the results across studies.

As a prespecified subgroup analysis, we also calculated the performance of the Bacterial Meningitis Score in the subgroup of study patients not included in either the internal9 or the multicentre10 validation studies conducted by this study's investigators.

We performed statistical calculations using Stata statistical software.24

Results

We identified 405 published studies using our search strategy of which 10 met our inclusion criteria. We excluded an adult cohort study (111 patients)25 and a paediatric cohort study (91 patients)26 because insufficient details were provided to allow us to assess the patient population or the antibiotic pretreatment status. We included the remaining eight studies in this analysis (table 2).9,,16 We assessed the quality of the included studies using the QUADAS-2 instrument (table 3).22

Table 2

Published Bacterial Meningitis Score validation studies

Table 3

Quality assessment of diagnostic accuracy studies 222

The eight included studies had the following study designs: case series (one study),13 case-control (one study),12 retrospective cohort (five studies)9,,11 ,14 ,16 and prospective cohort (one study).15 Study patients presented for emergency care in the USA,9 ,10 Western Europe11 ,12 and South America (Argentina).15 Of the 5312 included children, 1242 (23%) had bacterial meningitis and 4070 (77%) had aseptic meningitis.

The patient populations for each validation study varied slightly (table 4). One study was limited to patients with bacterial meningitis13 and another had available procalcitonin results.14 Five studies were of hospitalised children, including patients referred for management of meningitis.9 ,11 ,12 ,14,,16 One other study was conducted in the emergency department10 and the other was a nationwide meningitis registry (216 participating institutions).13 ,27 While Haemophilus influenzae type B vaccination rates were high in all study populations, only one population (USA) had widespread seven-valent pneumococcal conjugate vaccination available during the study period.10 ,28 None of the populations had routine meningococcal vaccination during the study period. All studies excluded children with immunosuppressive medical conditions or therapies, while the other exclusion criteria varied by study: critical illness,9 ,10 ,14,,16 recent neurosurgery or presence of a ventricular shunt,9 ,10 ,12,,16 purpura,9 ,10 focal bacterial infections requiring parenteral antibiotic treatment,9 ,10 traumatic LP,12 ,13 ,16 Lyme meningitis16 or transferred patients.16

Table 4

Study definitions

While in all studies the bacterial meningitis case definition included patients with CSF culture positive for a bacterial pathogen, they varied in whether patients with a positive CSF Gram stain,12 latex agglutination test,12 ,14,,16 CSF bacterial PCR test15 ,16 or CSF pleocytosis with a positive blood culture but negative CSF culture9,,11 ,13 ,15 ,16 were considered to have bacterial meningitis. Although patients pretreated with antibiotics were excluded, the time between antibiotic administration and diagnostic LP defined as antibiotic pretreatment was not standardised.

The Bacterial Meningitis Score could be calculated for 4896 (92%) of the 5312 patients in the aggregated patient population (table 5). Patients from all studies contributed to the calculation of test sensitivity. Patients from seven studies were included in the calculation of test specificity and from six studies in the predictive value and likelihood ratio calculations. The Bacterial Meningitis Score had an overall test sensitivity of 99.3% (95% CI 98.7 to 99.7%) (figure 1). For the primary outcome measure (sensitivity), we found no evidence for heterogeneity between studies (p=0.88) or results (I2=0.0) in our meta-analysis. Children categorised as ‘not very low risk’ by the Bacterial Meningitis Score had a positive likelihood ratio for bacterial meningitis of 2.6 (95% CI 2.5 to 2.7) and those categorised as ‘very low risk’ by the Bacterial Meningitis Score had a negative likelihood ratio of 0.01 (95% CI 0.01 to 0.02).

Figure 1

Forest plot showing Bacterial Meningitis Score sensitivity with 95% CI for individual studies (and a pooled estimate).

Table 5

Performance of the Bacterial Meningitis Score (BMS)

Among the 1783 children included in the 6 validation studies not conducted by this study's investigators, 1083 (61%) had bacterial meningitis and 700 (39%) had aseptic meningitis. In this subgroup, the Bacterial Meningitis Score had a sensitivity of 99.3% (95% CI 98.6% to 99.7%), specificity of 61.0% (95% CI 57.3 to 64.7%), NPV of 98.3% (95 CI 96.6% to 99.3%) and PPV 28.1% (95% CI 22.6% to 33.9%). We could not exclude the possibility that children with bacterial meningitis from 1 of the 216 centres in the bacterial meningitis registry13 may also have been included in a more recent multicentre retrospective cohort study (personal communication, M. Chalumeau, Hôpital de Paris). However, when we excluded this cohort study, the Bacterial Meningitis Score performed similarly (data not shown).

Nine patients with bacterial meningitis were classified as ‘very low risk’ by the Bacterial Meningitis Score (table 5). Of these, three were younger than 2 months of age (an age at which we previously recommended the Bacterial Meningitis Score not be applied)10 and three others presented with petechiae or purpura on examination.13 The three misclassified patients with bacterial meningitis who were older than 2 months and who did not have a petechial or purpuric rash on presenting examination are described in table 6 (patients 3, 6 and 9). All three of these children had meningococcal meningitis.

Table 6

Characteristics of children with bacterial meningitis who were misclassified as very low risk by the Bacterial Meningitis Score

Discussion

The Bacterial Meningitis Score, a clinical prediction rule to identify children with CSF pleocytosis who are at very low risk of bacterial meningitis, has been validated in eight published studies.9,,16 In this meta-analysis, the score had a sensitivity of 99.3% (95% CI 98.7% to 99.7%) for bacterial meningitis. The included studies were of high quality and did not have significant study heterogeneity. Although the study designs, patient population, inclusion and exclusion criteria as well as bacterial meningitis case definition differ between studies, the Bacterial Meningitis Score performed with a high degree of accuracy. When the validation studies conducted by this study's investigators were excluded, the results did not change.

A clinical prediction rule is a decision-making tool that combines history, physical examination and laboratory results to predict the probability of an outcome for an individual patient. Clinical prediction rules must be developed and validated according to rigorous methodological standards prior to widespread implementation.29,,31 While prospective rather than retrospective validation is typically preferred, because of the rarity of bacterial meningitis in high-income countries,5 such a validation is not readily feasible; accordingly, seven of the eight included studies were retrospective. We do not believe that the retrospective validation introduced important biases since the predictors are objective. Four of the five factors are laboratory values and one clinical factor (seizure at or prior to the time of initial presentation), should be reliably recorded in the medical records.32 ,33

For children younger than 18 years of age in the USA, the incidence of bacterial meningitis caused by H influenzae, Streptococcus pneumoniae, group B streptococcus (GBS), Neisseria meningitidis or Listeria monocytogenes declined 31% over the past decade.5 Despite these substantial declines in the overall incidence, the bacterial meningitis case death rate has remained unchanged at 7%; most affected children do not have predisposing medical conditions.5 Given the high mortality and morbidity, clinicians must still maintain a high index of suspicion for bacterial meningitis. Looking ahead, the increasing availability of the 13-valent S pneumoniae and quadrivalent N meningitidis conjugate vaccines in high-income countries will further decrease the incidence of bacterial meningitis. The Bacterial Meningitis Score can support clinician decision making by providing a highly accurate initial assessment of the risk of bacterial meningitis for children with CSF pleocytosis.

We recommend that the Bacterial Meningitis Score, as with other clinical prediction rules, be used to assist rather than replace clinical decision making.34 Although the score performs extremely well, it is highly unlikely to develop a clinical prediction rule with 100% accuracy in all patient populations.35 Of note, the misclassified patients who presented without petechia or purpura all had meningococcal meningitis which has been previously described to present without CSF pleocytosis.36 Nevertheless, children with a very low-risk Bacterial Meningitis Score are at such low risk of bacterial meningitis that they may be considered for outpatient management, potentially after the administration of a long-acting parenteral antibiotic. Currently, most children with CSF pleocytosis are hospitalised and given parenteral antibiotics in order to avoid missing the very few with bacterial meningitis. Application of the Bacterial Meningitis Score could substantially reduce unnecessary hospitalisation of children with aseptic meningitis, while still providing a safety margin by the administration of long-acting antibiotic prior to culture results. As we have previously recommended,10 however, the Bacterial Meningitis Score should not be applied to ill-appearing children, infants 2 months and younger, in whom the risk of bacterial meningitis is highest5 and to those with physical examinations suggestive of invasive bacterial infection (eg, those with petechiae or purpura).37 This approach will further reduce the risk of misclassification of children with bacterial meningitis.

Furthermore, biologic markers of inflammation have been investigated for their ability to discriminate between cases of bacterial and aseptic meningitis. For example, CSF lactate38 ,39 and serum procalcitonin40 ,41 are higher in children with bacterial than aseptic meningitis, although there is overlap in levels of these biomarkers. Diagnostic assays that use RNA expression to identify host response to specific pathogens are currently being studied in the clinical setting.42,,44 In the future, these novel assays might allow rapid identification of specific meningitis pathogens, or host responses associated with bacterial meningitis.

Our study has some limitations. First, as most of the validation studies were retrospective, we could not evaluate the general appearance of the patients, which plays an important role in clinical decision making. Certain clinical factors such as the presence of petechiae or purpura could not be evaluated in all studies. Nevertheless, the accuracy of the Bacterial Meningitis Score remained very high in each of the evaluated studies. Second, we were only able to include children who had sufficient clinical data to apply the Bacterial Meningitis Score from published validation studies. Given the number of patients evaluated in these studies and the objective nature of the variables in the score, it is unlikely that the model would have performed substantially differently in patients with missing variables. Third, we could not exclude the possibility that a few children with bacterial meningitis from a single centre may have been represented in two included studies.13 ,14 However, the prediction model performed similarly if this recent study was excluded (and therefore eliminating the possibility of patient redundancy). Fourth, we were unable to include patients from the case-control or registry studies in the calculations of NPV, PPV or likelihood ratios as these calculations require population prevalence. The variability between experimental definitions and methods may have reduced our ability to accurately combine the study patients. However, the similar performances of the model across a wide variety of clinical settings and patient populations as well as the lack of heterogeneity in the sensitivity and NPV estimates increase the generalisability of our findings. Furthermore, we recognise that there is a preponderance (almost two-thirds) of patients from a single validation study.10 To address a potential skewing effect, we performed an additional subgroup analysis after excluding studies conducted by this study's investigators, and found similar results to the main analysis. Finally, the Bacterial Meningitis Score does not predict the likelihood of other central nervous system infections such as herpes simplex virus, Lyme45,,47 or tuberculous meningitis. Therefore, this clinical prediction rule should be used in concert with careful clinical assessment of the patient, which would include consideration of these other important treatable infections.

Conclusions

In summary, the Bacterial Meningitis Score performed with a high degree of diagnostic accuracy in eight validation studies. This score, in conjunction with clinical judgment can identify children with CSF pleocytosis who are at very low risk for bacterial meningitis. To minimise misclassification of children with bacterial meningitis, we recommend that the Bacterial Meningitis Score only be applied to non-ill-appearing children older than 2 months, who do not have either petechiae or purpura on examination and have not been pretreated with antibiotics. For those children at very low risk, who have adequate clinical follow-up, clinicians could consider outpatient treatment after administration of a long-acting parenteral antibiotic. Future studies should focus on the implementation of the Bacterial Meningitis Score to prospectively identify children who are at very low risk of bacterial meningitis.

Acknowledgments

The authors would like to thank Michael C Monuteaux ScD (Division of Emergency Medicine, Children's Hospital; Boston, Massachusetts, USA) for his help with statistical analysis.

References

View Abstract

Footnotes

  • This work was presented in part at the American Academy of Pediatrics, National Conference and Exhibition, 14 October 2011, Boston, Massachusetts, USA.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.