You are here

Article Text

Article menu

Download PDFPDF

Original article
Clinical prediction models for young febrile infants at the emergency department: an international validation study
  1. Evelien de Vos-Kerkhof1,
  2. Borja Gomez2,3,
  3. Karen Milcent4,
  4. Ewout W Steyerberg5,
  5. Ruud Gerard Nijman6,
  6. Frank J Smit7,
  7. Santiago Mintegi2,3,
  8. Henriette A Moll1,
  9. Vincent Gajdos8,
  10. Rianne Oostenbrink1
  1. 1 Department of General Paediatrics, Erasmus MC-Sophia Children’s Hospital, Rotterdam, The Netherlands
  2. 2 Paediatric Emergency Department, Cruces University Hospital, Bilbao, Spain
  3. 3 University of the Basque Country, Bilbao, Spain
  4. 4 AP-HP Department of Paediatrics, Hôpitaux Universitaires Paris Sud–Antoine Béclère, Clamart, France
  5. 5 Department of Public Health and Clinical Decision Making, Erasmus MC–University Medical Centre Rotterdam, Rotterdam, The Netherlands
  6. 6 Department of Paediatric Accident and Emergency, St Mary’s Hospital, Imperial College–NHS Healthcare Trust, Rotterdam, The Netherlands
  7. 7 Department of General Paediatrics, Maasstad Hospital, Rotterdam, The Netherlands
  8. 8 Université Paris-Saclay, Université Paris-Sud, UVSQ, CESP, INSERM, Villejuif, France
  1. Correspondence to Dr Rianne Oostenbrink, Department of General Paediatrics, Erasmus MC-Sophia Children’s Hospital, Rotterdam 3015, The Netherlands; r.oostenbrink{at}erasmusmc.nl

Abstract

Objective To assess the diagnostic value of existing clinical prediction models (CPM; ie, statistically derived) in febrile young infants at risk for serious bacterial infections.

Methods A systematic literature review identified eight CPMs for predicting serious bacterial infections in febrile children. We validated these CPMs on four validation cohorts of febrile children in Spain (age <3 months), France (age <3 months) and two cohorts in the Netherlands (age 1–3 months and >3–12 months). We evaluated the performance of the CPMs by sensitivity/specificity, area under the receiver operating characteristic curve (AUC) and calibration studies.

Results The original cohorts in which the prediction rules were developed (derivation cohorts) ranged from 381 to 15 781 children, with a prevalence of serious bacterial infections varying from 0.8% to 27% and spanned an age range of 0–16 years. All CPMs originally performed moderately to very well (AUC 0.60–0.93). The four validation cohorts included 159–2204 febrile children, with a median age range of 1.8 (1.2–2.4) months for the three cohorts <3 months and 8.4 (6.0–9.6) months for the cohort >3–12 months of age. The prevalence of serious bacterial infections varied between 15.1% and 17.2% in the three cohorts <3 months and was 9.8% for the cohort >3–12 months of age. Although discriminative values varied greatly, best performance was observed for four CPMs including clinical signs and symptoms, urine dipstick analyses and laboratory markers with AUC ranging from 0.68 to 0.94 in the three cohorts <3 months (ranges sensitivity: 0.48–0.94 and specificity: 0.71–0.97). For the >3–12 months’ cohort AUC ranges from 0.80 to 0.89 (ranges sensitivity: 0.70–0.82 and specificity: 0.78–0.90). In general, the specificities exceeded sensitivities in our cohorts, in contrast to derivation cohorts with high sensitivities, although this effect was stronger in infants <3 months than in infants >3–12 months.

Conclusion We identified four CPMs, including clinical signs and symptoms, urine dipstick analysis and laboratory markers, which can aid clinicians in identifying serious bacterial infections. We suggest clinicians should use CPMs as an adjunctive clinical tool when assessing the risk of serious bacterial infections in febrile young infants.

  • epidemiology
  • evidence-based medicine
  • general paediatrics
  • infectious diseases

http://dx.doi.org/10.1136/archdischild-2017-314011

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    What is already known on this topic?

    • Febrile children, and young febrile children in particular, are at risk of serious bacterial infections, such as pneumonia, sepsis or meningitis, which are among the leading causes of morbidity and mortality in children both in developed and developing countries.

    • The management of febrile young infants is hampered by the difficulty of discriminating between children with serious bacterial infections and those with self-limiting infectious illnesses, as specific ‘red flag’ features are often absent.

    • Existing clinical prediction models for febrile children often are derived from small numbers of children aged <1 year, and few focused specifically on febrile children aged <1 year.

    What this study adds?

    • Validation using cohorts from three European emergency care settings found four out of eight clinical prediction models showed satisfactory performance in identifying serious bacterial infections in febrile infants, but favoured specificity above sensitivity.

    • Clinical prediction models including clinical signs and symptoms, urine dipstick analysis and laboratory markers had better performance characteristics in young febrile infants compared with clinical prediction models including only symptoms and signs.

    Introduction

    Fever is a major cause for seeking medical care in childhood.1 Febrile children are at risk of serious bacterial infections, such as pneumonia, sepsis or meningitis. Despite the low incidences of serious bacterial infections, their delayed diagnoses are among the leading causes of morbidity and mortality in children.2 3 Clinical decision-making in febrile infants is hampered by the inability to easily discriminate the minority of children with serious bacterial infections from those with self-limiting viral illnesses.4 5 This is particularly true for young (age <1 year) febrile children, as young infants have the physiological ability to retain cardiovascular and respiratory parameters within normal ranges for a prolonged period, even in the presence of a serious infection.6 Also specific ‘red flag’ signs of serious bacterial infections may not be present in young children (eg, meningeal signs) and we must rely on non-specific features (eg, lethargy).7 Moreover, young infants are more susceptible to serious bacterial infections as they will not have completed their routine childhood immunisation schedules.8

    In paediatric emergency care, low-risk criteria (Rochester9 or variations of these10–13) are often used to assist predicting which children are more likely to have serious bacterial infections. These criteria generally use a mix of general appearance, medical history, focal infection and white cell count to rule out serious bacterial infections (SBI). These consensus-based low-risk criteria date from an era with a much higher prevalence of SBI as a result of changing epidemiology, due to extended vaccination strategies, group b streptococcu screening and replacing emerging or seasonal pathogens.14–16

    Alongside these low-risk criteria, clinical prediction models (CPM) with a statistical rather than consensus basis have been introduced to support the diagnostic process.17 18 They provide physicians with estimates of the probability of a child having a serious infection by combining clinical signs and symptoms and additional diagnostic test results in mathematical models.

    Studies in which CPMs for children at risk for serious bacterial infections were derived often included only limited numbers of young children aged <1 year, and only few of these CPMs specifically targeted young infants.7 Hence, broad validation and implementation studies of CPMs for febrile children at risk of serious bacterial infections are an essential, yet currently missing, step in the translation of CPMs into clinical practice.19 20 As a result, the clinical value of existing CPMs remains unclear in this cohort of young infants at risk for serious bacterial infections. The aim of this study was to systematically search for CPMs targeting febrile children at risk of serious bacterial infections in emergency or primary care and to subsequently validate these CPMs in four different cohorts of febrile young infants aged <1 year from European emergency care settings.

    Methods

    Inclusion criteria systematic review

    We included studies that1: described a CPM with the purpose of predicting serious bacterial infections in febrile children2; included three or more predictors which were routinely determined on clinical examination, possibly extended by diagnostic test results3; advised on management strategies (ie, clinical decision model) or gave a risk score/classification (ie, clinical prediction rule)4; and were based on statistical modelling such as multivariable logistic regression analysis. We excluded studies that: (1) were derived using expert opinion or consensus-based approaches; (2) investigated one specific diagnosis only (eg, pneumonia, meningitis); (3) included biomarkers only; (4) included immune-compromised patients; (5) were performed in low-income countries; (6) were published before the year 2000; and (7) were not published in English.

    Search strategy and risk of bias assessment systematic review

    We systematically searched Medline OvidSP, Embase, CINAHL, Cochrane central register of controlled trials, Web of Science, Google Scholar and PubMed as publisher (searches updated in May 2016) (online supplemental information 1). We then checked the reference list of these papers for additional articles that were not found by the initial computerised search. Two authors (EVK/RO) independently assessed the potential assessed risk of bias of the studies included using the QUADAS list, a quality assessment tool of diagnostic accuracy studies.21 The tool is a list of 14 questions to be answered ‘yes’, ‘no’ or ‘unclear’ (online supplemental information 2). Studies selected for analysis were given an A, B, C or D rating according to a previous described scoring system.22 Consensus was reached by the two reviewers (EVK/RO) if there was disagreement on an item. If no consensus was achieved, the opinion of a third independent reviewer was decisive (HAM). Moreover, we added the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) checklist (online supplemental information 3), which aims to improve the reporting of studies developing, validating or updating a prediction model.23

    Supplemental material

    Supplemental material

    Supplemental material

    Data extraction and management validation study

    In this validation study we used four different cohorts of febrile young infants, including two cohorts with children <3 months from Spain (Cruces University Hospital, Bilbao, Basque Country, Spain) and France (15 French paediatric emergency departments (ED)),24 as well as one cohort with children aged <1 year from two hospitals in Rotterdam, the Netherlands (Maasstad Hospital and Sophia Children’s Hospital). The Dutch cohort was divided into a cohort of children aged <3 months (the Netherlands <3 months’ cohort) and children aged between 3 and <12 months (the Netherlands >3–12 months’ cohort). In the Dutch cohort we prospectively enrolled all children (1 month to 1 year) presenting with fever at the EDs of the Erasmus MC-Sophia Children’s Hospital (inner city university hospital, 7000 annual paediatric ED visits)25 and the Maasstad Hospital (inner-city teaching hospital, 10 000 paediatric medical emergency consults yearly), Rotterdam (2009–2012), the Netherlands. In both hospitals febrile children were eligible to participate in the study if fever had been noted at home in the 24 hours prior to presentation, if body temperature measured at the ED was ≥38.5°C or ‘fever’ was used as a positive discriminator of the Manchester Triage System. Children with chronic comorbidity were excluded. Informed consent was required and obtained. Detailed description of this febrile child cohort was published earlier.17 26 The Paediatric Emergency Department of Cruces University Hospital in Bilbao is a tertiary teaching hospital and provides care for approximately 55 000 children per year of less than 14 years of age. During 9 consecutive years (2003–2012) data on infants younger than 3 months of age with fever without source (FWS) seen in the ED were prospectively collected.27–31 FWS was defined as temperature measured at home or at the ED ≥38°C and without a clear source of the origin of the fever. The study was approved by the local Research Committee and as the registry contained anonymous information, and as no intervention was performed, the need for informed consent was waived.27 Lastly, the prospective cohort study of France was conducted at 15 paediatric EDs in several regions in France. These hospitals were both teaching and tertiary centres seeing 15 000–50 000 children annually. From October 2008 through March 2011 febrile children (≥38°C at home or on admission) aged between 7 and 91 days without major comorbidities were included.24 The local ethics committee approved the study and parental written informed consent was obtained.24 Definitions of serious bacterial infections were comparable in all four cohorts and included pneumonia, and ‘other’ serious bacterial infections as meningitis, septicaemia, urinary tract infections, erysipelas, cellulitis, bacterial gastroenteritis, orbital cellulitis, bacterial upper respiratory infection, ethmoiditis, septic arthritis and osteomyelitis. All definitions were defined by positive bacteriological cultures of normally sterile sites and/or by nodular infiltrates or consolidation in the lung on chest radiographs.17 24 27 All cohorts included a follow-up period to ensure no serious infections were missed.

    Statistical analyses

    When the selected CPMs included variables that were not identical to the variables available in our four cohorts we identified proxies by (1) combining available features into one predictor variable (eg, ‘dyspnoea’ and ‘chest-wall retractions’ into ‘shortness of breath’) or (2) dichotomising categorical variables (eg, mild, moderate, severe) into ‘absent ‘or ‘present’ (online supplemental information 4). In the situation when an original variable contained more categories (eg, mild, moderate, severe) than were available for a proxy variable (eg, absent, present) beta coefficients were recalculated as a weighted mean on the basis of the originally published beta coefficients and the number of patients with the variable present in the derivation cohort. If a CPM predictor variable was completely absent we used the prevalence observed in the original derivation cohort. Missing data of other predictor variables were assumed to be missing at random, and were imputed 10 times using a multiple imputation process with the mice algorithm in R software (V.3.0).32 Different imputational models were developed for each individual CPM, including CPM predictor variables, the outcome variable and variables describing case-mix of the cohort. Imputation was done per validation cohort and results were analysed using Rubin’s rules.33

    Supplemental material

    We only validated the CPMs if  >2/3 of the required variables were available in the validation data sets.33 The performance of CPMs in allocating children to high-risk versus low-risk categories of serious infection was evaluated by calculating sensitivity, specificity and positive/negative likelihood ratios (LR+ and LR−). Area under the receiver operating characteristic curve (AUC) was used to determine discriminative ability. For CPMs that provided risk predictions on a continuous (0%–100%) scale we assessed calibration graphically and we compared the incidence of cases of serious bacterial infections with the average predicted risk in each validation cohort (calibration in the large). In addition, the calibration slope was calculated using a logistic regression model with the log odds of predicted risk as the only covariate (logistic calibration).34 35 Logistic calibration with a slope of 1 means perfect calibration with similar predictive effects of the model in both the derivation and validation cohorts. Analyses were performed with SPSS software (V.20.0, IBM).

    Results

    Identification and description of CPMs

    The literature search identified 2549 references. Figure 1 presents the flow chart of papers screened for eligibility. Data extraction was performed for eight included articles, each describing a single CPM.17 18 36–41 Seven CPMs were derived in secondary care settings and one CPM41 was derived in primary care (table 1). The prevalence of serious bacterial infections in the original study cohorts varied between 0.8% and 27%, and included between 381 and 15 781 children who had median ages ranging from 0.1 to 0.6 years. Combined, the 8 CPMs included 35 unique clinical variables (range 3–28 variables), with age, temperature, general appearance, capillary refill time, respiratory rate, breathing difficulty and urine dipstick analyses being the most frequently reported ones (online supplemental information 5). Four of the CPMs expressed the risk for serious bacterial infections in a dichotomous way (high/low)36 38 39 41 whereas the other four CPM predicted risks were on a continuous scale (0%–100%) (table 1).17 18 37 40

    Figure 1
    Figure 1

    Flow chart systematic review of clinical prediction models for determination of serious bacterial infections.

    View this table:
    Table 1

    Overview of selected clinical prediction models*

    Risk of bias assessment

    All studies failed to report data on at least one or more QUADAS criteria (online supplemental information 2). The CPM by Bachur and Harper (CPM 1)36 and Galetto-Lacour et al (CPM 3)38 had the lowest risk of bias. There was initial disagreement on 8 out of the 112 assessed QUADAS items (7%), consensus was reached for all these items by the two reviewers (EVK/RO) and no third opinion was required.

    Description of validation cohorts

    The Dutch cohort consisted of 925 febrile children under the age of 1 year17 of whom 159 (17.2%) were under the age of 3 months. The Spanish cohort included 2148 febrile children aged <3 months, the French cohort 2204 children.24 27 The inclusion flow charts of these validation cohorts were published earlier.17 24 27 The prevalence of serious bacterial infections varied between 15.1% and 17.2% in the three cohorts of febrile children aged <3 months and was 9.8% for the cohort >3–12 months of age (table 2). A number of original CPM predictor variables were absent in some or all validation cohorts. Most notably this was the case for vital signs (missing in the Spanish cohort) and specific respiratory symptoms (missing in all cohorts) (online supplemental information 5).

    View this table:
    Table 2

    Patient characteristics of the validation cohorts

    Performance of included CPMs

    We evaluated the performance of seven CPMs in the Dutch cohorts, four CPMs in the Spanish cohort and all eight CPMs in the French cohort. Two36 38 out of four CPMs36 38 39 41 with high/low-risk classifications performed reasonably well with the discriminative ability (AUC) ranging from 0.72 to 0.87 in all three validation cohorts <3 months. Corresponding sensitivities ranged from 0.48 to 0.94 and specificities from 0.80 to 0.97 (table 3). These two CPMs also showed moderate discriminative values in the cohort >3–12 months.36 38

    View this table:
    Table 3

    Performance of clinical prediction models with high/low-risk prediction

    Of the four CPMs using continuous risk predictions17 18 37 40 CPM5 (37) was validated in all cohorts with AUC ranging from 0.78 to 0.94 in the three cohorts of children <3 months and AUC 0.82 in the cohort of children >3–12 months. CPM 8 (17) could not be validated in the Spanish cohort but achieved good performance in both the two Dutch cohorts and the French cohort to predict pneumonia (range AUC 0.72–0.89), and to predict other SBI (range AUC 0.68–0.82) (table 4).

    View this table:
    Table 4

    Performance of clinical prediction models with continuous risk prediction

    All CPMs calculating continuous risk predictions (CPM 5–8)17 18 37 40 showed poor calibration in the large. Logistic calibration showed almost equal predictor effects as at development for CPM 5 (42) in the two Dutch and the French validation cohorts (slope 0.97–1.15), but underestimated predictor effects in the Spanish cohort (slope 2.95, SE 2.81–3.09). For CPM 8 (17) we had contradictory findings in both Dutch cohorts, with underestimated predictor effects for pneumonia in children <1 year and overestimated predictor effects for other SBI. In the French cohort (<3 months) we observed overestimated predictor effects in both outcomes.

    Discussion

    Main findings

    Eight CPMs were successfully validated in more than 4500 febrile young infants from three European emergency care cohorts. Four out of eight validated CPMs classifying young febrile infants into high-risk versus low-risk categories achieved consistently satisfactory performance for infants <3 months. These four CPMs are therefore potentially suitable in guiding medical decision-making in young febrile infants at risk of serious bacterial infections. Although the specificities of the CPMs generally exceeded the sensitivities of the CPMs in young infants, the overall discriminative ability (as defined by the AUC) in these best performing CPMs was comparable. We observed that CPMs including a combination of clinical signs and symptoms (temperature, general appearance, capillary refill time and breathing difficulty), urine dipstick analyses and laboratory markers (white blood cell (WBC) and C-reactive protein (CRP)) performed better in young infants than CPMs with just clinical parameters. Performance of CPM in infants 3–12 months were quite similar to the performance of CPMs in the cohorts with children aged <3 months.

    Comparison with literature

    We found general appearance to be a consistent and important clinical predictor variable for identifying serious bacterial infections in febrile young infants. This is supported by, for example, the accuracy study of Pantell et al in which ill appearance was one of the strongest clinical predictors of bacteraemia or bacterial meningitis in infants less than 3 months old.42 Also, the importance of combining biomarkers and clinical signs and symptoms to ensure validity of CPMs was underlined by several other studies.17 22 43–45 Moreover, biomarkers outperformed the clinician’s risk assessment, as generated by visual analogue scores of having SBI.46 Furthermore, the important diagnostic role of urine dipstick analyses was previously demonstrated by De et al who, based on the study’s findings, showed improved diagnostic performance of the National Institute for Health and Care Excellence (NICE) traffic light system recommended adding urine dipstick analysis to the NICE traffic light system.47

    To better interpret the observed diagnostic performance of the validated CPMs based on statistically modelling in this study, it might be worthwhile to compare them with consensus-based guidelines or expert opinion-based criteria such as the step-by-step approach of Mintegi et al or the well-known low-risk criteria of Rochester9 and its variations.10–13 For example, Huppler et al 48 concluded that the low-risk criteria performed well, supporting the decision to safely withhold antibiotic treatment with a watchful waiting approach in young febrile infants <3 months. Applying these low-risk criteria of Rochester to our validation cohorts (n=5277) allocated 2033 (39%) children into low risk, with a false-positive rate (risk criteria present, but no SBI diagnosed) of 47% (2479/5277), and 66 SBI cases were classified as low risk by Rochester criteria (1%). Defining the high-risk cohort by the step-by-step criteria (ill appearance; ≤21 days; leukocyturia; procalcitonine ≥0.5) resulted in sensitivities varying from 0.78 to 0.94 in our cohorts against specificities ranging from 0.63 to 0.80. We would like to advocate the preferred use of our identified four CPMs17 36–38 as they showed reasonable performance and much lower false-positive rates for serious bacterial infections in this external validation study.

    Interpretation of results and clinical implications

    As the cost of missing a serious bacterial infection in a young febrile infant is potentially severe, we are in need of CPMs with high rule-out value in our daily practice (high sensitivity and strong negative LR). The four CPMs (CPMs 1, 3, 5 and 8) achieving the best performance in this study were all developed in secondary care. These CPMs are potentially suitable for identifying serious bacterial infections in young febrile infants. However, their sensitivity is not consistently high in all validation settings, although not consistently related to age.

    Logistic calibration studies showed that most predictor effects were underestimated, that is, the predictor effects are probably stronger in these younger cohorts compared with the original cohorts. However, not all differences in validation performance are likely to be the result of this, as calibration in the large showed differences in (unobserved) case-mix in derivation and validation cohorts. We observed that CPM 1 (36) and CPM 5 (37) performed particularly well in the Spanish cohort, potentially due to similarities between derivation and validation cohorts. Many diagnostic studies exclude children <1 month, but the derivation cohort of CPM 1 (36) only included young infants <3 months. Next, the derivation cohort of CPM 5 (37) also targeted febrile children without a source, although including a much wider age range than the cohort at interest in our validation study. One could argue that performance of the rules in our validation study merely relates to differences in cohort and case-mix in general, rather than to the age difference. CPM 2 (41), however, has been externally validated in hospital cohorts elsewhere showing consistent sensitivity but with lower specificity. This contrast with our results, showing a relatively large reduction in sensitivity but only a minor loss of specificity.49 It should be noted that our focus on young febrile children reduces age effects in all CPMs that were developed for children of a much wider age range anyway. For example, this may have affected CPM 2 in particular,41 as this CPM included two cut-off points of age >1 year and which was therefore not applicable to our young infants aged <1 year. Likewise, even though CPM 4 (39) was derived in a cohort of infants >3 months of age, we observed similar performances in the Dutch cohort between the infants <3 months and the infants 3–12 months. A closer look at the predictors of the best performing CPMs in at least three validation cohorts (CPMs 1, 3, 5 and 8) showed a more consistent role of the clinical predictors’ general appearance, temperature, capillary refill time, duration of fever and breathing difficulty. Moreover, these CPMs included laboratory markers as CRP, WBC and urine dipstick analyses. The performance of CPM 3 (38) and CPM 8 (17) including general appearance and/or laboratory markers is comparable in all our four cohorts with their performances in the derivation cohorts. In contrast, CPMs not including general appearance and/or laboratory markers (CPMs 2, 4 and 6) performed worse than in their derivation cohorts. This highlights the importance of laboratory markers as adjuncts to clinical signs and symptoms for the identification of serious bacterial infections in young febrile infants. However, no existing CPM is able to identify all relevant cases of serious bacterial infection in clinical practice, highlighting the position of CPM as tools in assisting clinical decision-making.

    Unfortunately, we were able to validate CPM 7, a polytomous model consisting of 27 clinical variables in only one cohort because of the absence of too many predictor variables in the other cohorts. In the French cohort, this model performed particularly well for the outcome ‘pneumonia’. However, compliance with a complex model containing many predictor variables will be limited in clinical practice.50–53 Moreover, the value of CPM 7 (18) is limited for children with FWS, because many predictor variables include symptoms related to a focus of infection (eg, wheeze, stridor, abnormal ear/nose/throat signs) that will be absent in cohorts of very young aged children with non-specific symptoms.

    In summary, it remains difficult to identify one optimal CPM for young infants. For example, CPM 3 (38) shows a consistent high specificity in all validation cohorts, but could also result in considerable overtreatment due to its low sensitivity. Application of CPM 8 seems to require setting specific recalibration studies prior to implementation. We suggest that the use of robust CPMs is important in clinical practice evaluating febrile infants. They appear to outperform currently used low-risk criteria. However, given all the models’ fairly low sensitivity, using CPMs in clinical practice should be considered with some caution and only as an adjunct to the diagnostic work-up of febrile infants. Choices for the optimal CPM may be centre specific depending on setting.

    Strengths and limitations

    A major strength of our study is a synthesis of four large cohorts of febrile children with variable backgrounds to assess the diagnostic performance of existing CPMs. In addition, we focused on young children in particular, whereas most studies on CPMs focused on broader age categories. Moreover, we evaluated all potentially relevant CPMs by systematically reviewing the literature.

    This study also has limitations. We included CPMs derived in cohorts with low SBI prevalence (from primary care) to high prevalence settings of febrile children without a source.37 38 We performed calibration studies to study systematic errors in risk prediction and overall predictor effects, but did not study the diagnostic value of each individual CPM variable. However, we feel this has been done exhaustively in previous work, such as a systematic review by Van den Bruel et al.22 Second, despite the considerable size of the validation cohorts, there were only a limited number of children with pneumonia in the Spanish cohort (n=4) and bacteraemia (n=1 in the <3 months’ cohort and n=6 for the cohort >3–12 months) in the Dutch cohort, thus limiting the precision of validating CPM 7 (18) and CPM 8 (17). Furthermore, not all CPM predictor variables had been recorded in all validation cohorts, necessitating the use of proxy variables or using the original prevalence data for completely absent variables. These original prevalence data were mostly used in the CPM of Craig et al (CPM 7) due to the great number of variables used in this CPM. However, this CPM could only be validated in the French cohort. Next, given the small number of the Dutch <3 months’ cohort (n=159, 15.1% SBI) results of this cohort should be interpreted with care, although the results were comparable with the larger validation cohorts. Pooled analysis of cohorts was considered, but not performed for the observed substantial differences in diagnostic performances between cohorts. Apparently, although similar inclusion criteria were used and similar SBI prevalence observed in the validation cohorts, there was still significant heterogeneous case-mix, as illustrated by the calibration studies.54 This may in particular have affected the validation of CPM 7 with both a substantial difference in age distribution, prevalence and distribution of diagnoses.

    Although this study did not aim to develop a new prediction rule, the results pose the question whether or not there is a need for other prediction rules with more age-specific predictors. Based on previous literature and current findings, there did not appear to be clinical predictors specifically for young febrile infants. Larger studies could look into this into more detail, but it might be that biomarker discovery studies, using novel biomarkers or proteomics platforms, might prove more fruitful.

    Last, most of the included studies had a low QUADAS quality score. This was mostly inherent to the criteria of the verification of the outcome and performance of reference tests. This was also true for the validation cohorts. However, to minimise the risk of verification bias, all studies included a follow-up period to reduce the chance of missed serious infection as has been used in other diagnostic studies in this area.17 18 Two of the best performing CPMs (CPM 1 (36) and CPM 3 (38)) achieved the lowest risk of bias assessment in the QUADAS quality score.

    Conclusion

    This study provides timely evidence for the diagnostic role of CPMs in identifying serious bacterial infections in febrile young infants. Four CPMs, including the combination of clinical signs and symptoms, urine dipstick analysis and laboratory markers, are best suited to aid physicians in identifying serious bacterial infections, favouring specificity above sensitivity. We did not observe differences in performance of CPMs in children <3 months compared with those aged 3–12 months. However, limited sensitivities of the CPM observed in all validation cohorts could lead to missing serious bacterial infections in this cohort of young febrile infants. We advocate clinicians to use CPMs as an adjunct clinical tool when assessing the risk of SBI in febrile young infants.

    Supplemental material

    Supplemental material

    References

    1. ↵
      2. Armon K ,
      3. Stephenson T ,
      4. Gabriel V , et al
      . Determining the common medical presenting problems to an accident and emergency department. Arch Dis Child 2001;84:390–2.doi:10.1136/adc.84.5.390
      OpenUrlAbstract/FREE Full Text
    2. ↵
      2. Veldhoen ES ,
      3. Wolfs TF ,
      4. van Vught AJ
      . Changes in infectious disease mortality among children in the Netherlands. Eur J Pediatr 2009;168:465–8.doi:10.1007/s00431-008-0772-4
      OpenUrlPubMed
    3. ↵
      2. Sidebotham P ,
      3. Fraser J ,
      4. Covington T , et al
      . Understanding why children die in high-income countries. Lancet 2014;384:915–27.doi:10.1016/S0140-6736(14)60581-X
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      2. Baraff LJ ,
      3. Lee SI
      . Fever without source: management of children 3 to 36 months of age. Pediatr Infect Dis J 1992;11:146–51.
      OpenUrlPubMedWeb of Science
    5. ↵
      2. Brook I
      . Unexplained fever in young children: how to manage severe bacterial infection. BMJ 2003;327:1094–7.doi:10.1136/bmj.327.7423.1094
      OpenUrlFREE Full Text
    6. ↵
      APLS. Advanced paediatric life support: the practical approach. London: BMJ Books, 2004.
    7. ↵
      2. Oostenbrink R ,
      3. Thompson M ,
      4. Steyerberg EW
      . Barriers to translating diagnostic research in febrile children to clinical practice: a systematic review. Arch Dis Child 2012;97:667–72.doi:10.1136/archdischild-2011-300667
      OpenUrlAbstract/FREE Full Text
    8. ↵
      RIVM. Rijksvaccinatieprogramma:vaccinatieschema, 2013.
    9. ↵
      2. Dagan R ,
      3. Powell KR ,
      4. Hall CB , et al
      . Identification of infants unlikely to have serious bacterial infection although hospitalized for suspected sepsis. J Pediatr 1985;107:855–60.doi:10.1016/S0022-3476(85)80175-X
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      2. Dagan R ,
      3. Sofer S ,
      4. Phillip M , et al
      . Ambulatory care of febrile infants younger than 2 months of age classified as being at low risk for having serious bacterial infections. J Pediatr 1988;112:355–60.doi:10.1016/S0022-3476(88)80312-3
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      2. Baker MD ,
      3. Bell LM ,
      4. Avner JR
      . Outpatient management without antibiotics of fever in selected infants. N Engl J Med 1993;329:1437–41.doi:10.1056/NEJM199311113292001
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      2. Baskin MN ,
      3. O’Rourke EJ ,
      4. Fleisher GR
      . Outpatient treatment of febrile infants 28 to 89 days of age with intramuscular administration of ceftriaxone. J Pediatr 1992;120:22–7.doi:10.1016/S0022-3476(05)80591-8
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      2. Chiu CH ,
      3. Lin TY ,
      4. Bullard MJ
      . Application of criteria identifying febrile outpatient neonates at low risk for bacterial infections. Pediatr Infect Dis J 1994;13:946–9.doi:10.1097/00006454-199411000-00002
      OpenUrlPubMedWeb of Science
    14. ↵
      2. Thigpen MC ,
      3. Whitney CG ,
      4. Messonnier NE , et al
      . Bacterial meningitis in the United States, 1998-2007. N Engl J Med 2011;364:2016–25.doi:10.1056/NEJMoa1005384
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      2. Gladstone RA ,
      3. Jefferies JM ,
      4. Faust SN , et al
      . Continued control of pneumococcal disease in the UK - the impact of vaccination. J Med Microbiol 2011;60:1–8.doi:10.1099/jmm.0.020016-0
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      2. Chandran A ,
      3. Watt JP ,
      4. Santosham M
      . Prevention of Haemophilus influenzae type b disease: past success and future challenges. Expert Rev Vaccines 2005;4:819–27.doi:10.1586/14760584.4.6.819
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      2. Nijman RG ,
      3. Vergouwe Y ,
      4. Thompson M , et al
      . Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study. BMJ 2013;346:f1706.doi:10.1136/bmj.f1706
      OpenUrlAbstract/FREE Full Text
    18. ↵
      2. Craig JC ,
      3. Williams GJ ,
      4. Jones M , et al
      . The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses. BMJ 2010;340:c1594.doi:10.1136/bmj.c1594
      OpenUrlAbstract/FREE Full Text
    19. ↵
      2. de Vos-Kerkhof E ,
      3. Nijman RG ,
      4. Vergouwe Y , et al
      . Impact of a clinical decision model for febrile children at risk for serious bacterial infections at the emergency department: a randomized controlled trial. PLoS One 2015;10:e0127620.doi:10.1371/journal.pone.0127620
    20. ↵
      2. Roukema J ,
      3. Steyerberg EW ,
      4. van der Lei J , et al
      . Randomized trial of a clinical decision support system: impact on the management of children with fever without apparent source. J Am Med Inform Assoc 2008;15:107–13.doi:10.1197/jamia.M2164
      OpenUrlCrossRefPubMed
    21. ↵
      2. Whiting P ,
      3. Rutjes AW ,
      4. Reitsma JB , et al
      . The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003;3:25.doi:10.1186/1471-2288-3-25
      OpenUrlCrossRefPubMed
    22. ↵
      2. Van den Bruel A ,
      3. Haj-Hassan T ,
      4. Thompson M , et al
      . Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet 2010;375:834–45.doi:10.1016/S0140-6736(09)62000-6
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      2. Moons KG ,
      3. Altman DG ,
      4. Reitsma JB , et al
      . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1–73.doi:10.7326/M14-0698
      OpenUrlCrossRefPubMed
    24. ↵
      2. Milcent K ,
      3. Gajdos V
      . Use of procalcitonin assays to predict serious bacterial infection in young febrile infants-reply. JAMA Pediatr 2016;170:623.doi:10.1001/jamapediatrics.2016.0385
      OpenUrl
    25. ↵
      2. Seiger N ,
      3. van Veen M ,
      4. Steyerberg EW , et al
      . Accuracy of triage for children with chronic illness and infectious symptoms. Pediatrics 2013;132:e1602–8.doi:10.1542/peds.2013-1076
      OpenUrlAbstract/FREE Full Text
    26. ↵
      2. Nijman RG ,
      3. Moll HA ,
      4. Vergouwe Y , et al
      . C-reactive protein bedside testing in febrile children lowers length of stay at the emergency department. Pediatr Emerg Care 2015;31:633–9.doi:10.1097/PEC.0000000000000466
      OpenUrlCrossRefPubMed
    27. ↵
      2. Mintegi S ,
      3. Benito J ,
      4. Sanchez J , et al
      . Predictors of occult bacteremia in young febrile children in the era of heptavalent pneumococcal conjugated vaccine. Eur J Emerg Med 2009;16:199–205.doi:10.1097/MEJ.0b013e32831cefc9
      OpenUrlCrossRefPubMed
    28. ↵
      2. Garcia S ,
      3. Mintegi S ,
      4. Gomez B , et al
      . Is 15 days an appropriate cut-off age for considering serious bacterial infection in the management of febrile infants? Pediatr Infect Dis J 2012;31:455–8.doi:10.1097/INF.0b013e318247b9f2
      OpenUrlCrossRefPubMed
    29. ↵
      2. Gómez B ,
      3. Mintegi S ,
      4. Benito J , et al
      . Blood culture and bacteremia predictors in infants less than three months of age with fever without source. Pediatr Infect Dis J 2010;29:43–7.doi:10.1097/INF.0b013e3181c6dd14
      OpenUrlCrossRefPubMed
    30. ↵
      2. Gomez B ,
      3. Mintegi S ,
      4. Rubio MC , et al
      . Clinical and analytical characteristics and short-term evolution of enteroviral meningitis in young infants presenting with fever without source. Pediatr Emerg Care 2012;28:518–23.doi:10.1097/PEC.0b013e3182587d47
      OpenUrlCrossRefPubMed
    31. ↵
      2. Gomez B ,
      3. Mintegi S ,
      4. Lopez E , et al
      . Diagnostic value of leukopenia in young febrile infants. Pediatr Infect Dis J 2012;31:92–5.doi:10.1097/INF.0b013e3182337ddb
      OpenUrlCrossRefPubMed
    32. ↵
      Team RdC eRFfSC. R: a language and environment for statistical computing, 2006.
    33. ↵
      2. Vergouwe Y ,
      3. Royston P ,
      4. Moons KG , et al
      . Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 2010;63:205–14.doi:10.1016/j.jclinepi.2009.03.017
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      2. Van Calster B ,
      3. Nieboer D ,
      4. Vergouwe Y , et al
      . A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016;74:167–76.doi:10.1016/j.jclinepi.2015.12.005
      OpenUrlCrossRefPubMed
    35. ↵
      2. Oostenbrink R ,
      3. Thompson M ,
      4. Lakhanpaul M , et al
      . Children with fever and cough at emergency care: diagnostic accuracy of a clinical model to identify children at low risk of pneumonia. Eur J Emerg Med 2013;20:273–80.doi:10.1097/MEJ.0b013e32835771fd
      OpenUrlCrossRefPubMed
    36. ↵
      2. Bachur RG ,
      3. Harper MB
      . Predictive model for serious bacterial infections among infants younger than 3 months of age. Pediatrics 2001;108:311–6.doi:10.1542/peds.108.2.311
      OpenUrlAbstract/FREE Full Text
    37. ↵
      2. Bleeker SE ,
      3. Derksen-Lubsen G ,
      4. Grobbee DE , et al
      . Validating and updating a prediction rule for serious bacterial infection in patients with fever without source. Acta Paediatr 2007;96:100–4.doi:10.1111/j.1651-2227.2006.00033.x
      OpenUrlPubMedWeb of Science
    38. ↵
      2. Galetto-Lacour A ,
      3. Zamora SA ,
      4. Andreola B , et al
      . Validation of a laboratory risk index score for the identification of severe bacterial infection in children with fever without source. Arch Dis Child 2010;95:968–73.doi:10.1136/adc.2009.176800
      OpenUrlAbstract/FREE Full Text
    39. ↵
      2. Thompson M ,
      3. Coad N ,
      4. Harnden A , et al
      . How well do vital signs identify children with serious infections in paediatric emergency care? Arch Dis Child 2009;94:888–93.doi:10.1136/adc.2009.159095
      OpenUrlAbstract/FREE Full Text
    40. ↵
      2. Brent AJ ,
      3. Lakhanpaul M ,
      4. Thompson M , et al
      . Risk score to stratify children with suspected serious bacterial infection: observational cohort study. Arch Dis Child 2011;96:361–7.doi:10.1136/adc.2010.183111
      OpenUrlAbstract/FREE Full Text
    41. ↵
      2. Van den Bruel A ,
      3. Aertgeerts B ,
      4. Bruyninckx R , et al
      . Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract 2007;57:538–46.
      OpenUrlAbstract/FREE Full Text
    42. ↵
      2. Pantell RH ,
      3. Newman TB ,
      4. Bernzweig J , et al
      . Management and outcomes of care of fever in early infancy. JAMA 2004;291:1203–12.doi:10.1001/jama.291.10.1203
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      2. Nijman RG ,
      3. Moll HA ,
      4. Smit FJ , et al
      . C-reactive protein, procalcitonin and the lab-score for detecting serious bacterial infections in febrile children at the emergency department: a prospective observational study. Pediatr Infect Dis J 2014;33:e273–9.doi:10.1097/INF.0000000000000466
      OpenUrlCrossRefPubMed
    44. ↵
      2. Luaces-Cubells C ,
      3. Mintegi S ,
      4. García-García JJ , et al
      . Procalcitonin to detect invasive bacterial infection in non-toxic-appearing infants with fever without apparent source in the emergency department. Pediatr Infect Dis J 2012;31:645–7.doi:10.1097/INF.0b013e31824dacf4
      OpenUrlCrossRefPubMed
    45. ↵
      2. Gomez B ,
      3. Bressan S ,
      4. Mintegi S , et al
      . Diagnostic value of procalcitonin in well-appearing young febrile infants. Pediatrics 2012;130:815–22.doi:10.1542/peds.2011-3575
      OpenUrlAbstract/FREE Full Text
    46. ↵
      2. Lacroix L ,
      3. Manzano S ,
      4. Vandertuin L , et al
      . Impact of the lab-score on antibiotic prescription rate in children with fever without source: a randomized controlled trial. PLoS One 2014;9:e115061.doi:10.1371/journal.pone.0115061
    47. ↵
      2. De S ,
      3. Williams GJ ,
      4. Hayen A , et al
      . Accuracy of the "traffic light" clinical decision rule for serious bacterial infections in young children with fever: a retrospective cohort study. BMJ 2013;346:f866.doi:10.1136/bmj.f866
      OpenUrlAbstract/FREE Full Text
    48. ↵
      2. Huppler AR ,
      3. Eickhoff JC ,
      4. Wald ER
      . Performance of low-risk criteria in the evaluation of young infants with fever: review of the literature. Pediatrics 2010;125:228–33.doi:10.1542/peds.2009-1070
      OpenUrlAbstract/FREE Full Text
    49. ↵
      2. Verbakel JY ,
      3. Van den Bruel A ,
      4. Thompson M , et al
      . How well do clinical prediction rules perform in identifying serious infections in acutely ill children across an international network of ambulatory care datasets? BMC Med 2013;11:10.doi:10.1186/1741-7015-11-10
      OpenUrlCrossRefPubMed
    50. ↵
      2. Stiell IG ,
      3. Wells GA
      . Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 1999;33:437–47.doi:10.1016/S0196-0644(99)70309-4
      OpenUrlCrossRefPubMedWeb of Science
    51. ↵
      2. Reilly BM ,
      3. Evans AT
      . Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006;144:201.doi:10.7326/0003-4819-144-3-200602070-00009
      OpenUrlCrossRefPubMedWeb of Science
    52. ↵
      2. Grol R ,
      3. Dalhuijsen J ,
      4. Thomas S , et al
      . Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ 1998;317:858–61.doi:10.1136/bmj.317.7162.858
      OpenUrlAbstract/FREE Full Text
    53. ↵
      2. Cabana MD ,
      3. Rand CS ,
      4. Powe NR , et al
      . Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999;282:1458–65.
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      2. Van Hoorde K ,
      3. Vergouwe Y ,
      4. Timmerman D , et al
      . Assessing calibration of multinomial risk prediction models. Stat Med 2014;33:2585–96.doi:10.1002/sim.6114
      OpenUrlCrossRefPubMed

    Footnotes

    • Contributors EdVK: conceptualised and designed the study, was responsible for data collection at one of the three sites, carried out the initial analyses, drafted the initial manuscript and approved the final manuscript as submitted. BG, KM, RGN, FJS, SM, VG: coordinated and supervised data collection at one of the three sites, critically reviewed the manuscript and approved the final manuscript as submitted. EWS: supervised the analyses, reviewed and revised the manuscript, and approved the final manuscript as submitted. HAM: conceptualised and designed the study, supervised data collection at one of the three sites, reviewed and revised the manuscript, and approved the final manuscript as submitted. RO: conceptualised and designed the study, supervised data collection at one of the three sites, supervised the analyses, reviewed and revised the manuscript, and approved the final manuscript as submitted. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Disclaimer To our best knowledge this article is not accessible as full paper (pdf), only the reference (title) can be retrieved from https://repub.eur.nl/pub/93311, the online e-publication of the PhD thesis: Integrating Clinical Decision Making and Patient Care at the Paediatric Emergency Department–focusing on children with serious bacterial infections. Erasmus University Rotterdam. EdVK (2016 September 20).

    • Competing interests None declared.

    • Patient consent Parental/guardian consent obtained.

    • Ethics approval The study was approved by the local institutional medical ethics committee.

    • Provenance and peer review Not commissioned; externally peer reviewed.