Article Text
PDF
Abstract
Objectives To develop and cross-validate a multivariable clinical prediction model to identify invasive bacterial infections (IBI) and to identify patient groups who might benefit from new biomarkers.
Design Prospective observational study.
Setting 12 emergency departments (EDs) in 8 European countries.
Patients Febrile children aged 0–18 years.
Main outcome measures IBI, defined as bacteraemia, meningitis and bone/joint infection. We derived and cross-validated a model for IBI using variables from the Feverkidstool (clinical symptoms, C reactive protein), neurological signs, non-blanching rash and comorbidity. We assessed discrimination (area under the receiver operating curve) and diagnostic performance at different risk thresholds for IBI: sensitivity, specificity, negative and positive likelihood ratios (LRs).
Results Of 16 268 patients, 135 (0.8%) had an IBI. The discriminative ability of the model was 0.84 (95% CI 0.81 to 0.88) and 0.78 (95% CI 0.74 to 0.82) in pooled cross-validations. The model performed well for the rule-out threshold of 0.1% (sensitivity 0.97 (95% CI 0.93 to 0.99), negative LR 0.1 (95% CI 0.0 to 0.2) and for the rule-in threshold of 2.0% (specificity 0.94 (95% CI 0.94 to 0.95), positive LR 8.4 (95% CI 6.9 to 10.0)). The intermediate thresholds of 0.1%–2.0% performed poorly (ranges: sensitivity 0.59–0.93, negative LR 0.14–0.57, specificity 0.52–0.88, positive LR 1.9–4.8) and comprised 9784 patients (60%).
Conclusions The rule-out threshold of this model has potential to reduce antibiotic treatment while the rule-in threshold could be used to target treatment in febrile children at the ED. In more than half of patients at intermediate risk, sensitive biomarkers could improve identification of IBI and potentially reduce unnecessary antibiotic prescriptions.
- epidemiology
- therapeutics
Data availability statement
An anonymized data set containing individual participant data is available in a public data repository: https://data.hpc.imperial.ac.uk/resolve/?doi=7549. DOI: 10.14469/hpc/7549. For inquiries to obtain the full dataset, please contact the data manager of the PERFORM consortium (Tisham.de08@imperial.ac. uk).
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
What is already known on this topic?
In children, distinction between invasive bacterial and self-limiting infections on only clinical symptoms is unreliable leading to overuse of antibiotics on the one hand, but to missed invasive bacterial infections in others.
Several clinical prediction models including biomarkers have been developed to help decision making by risk prediction of patients at high risk or low risk for bacterial infections, but none predicts the outcome invasive bacterial infections in older children or includes children with chronic conditions.
What this study adds?
We derived and externally validated a clinical prediction model based on clinical predictors from the Feverkidstool (clinical symptoms, C reactive protein) and non-blanching rash, neurological symptoms and comorbidity, to early recognise invasive bacterial infections with data from a large observational European-wide study of febrile children aged 0–18 years.
The rule-out threshold of this model could reduce antibiotic prescription and invasive diagnostics, while the rule-in threshold could be useful to target early treatment for invasive bacterial infections.
In more than half of the patients at intermediate risk, sensitive new biomarkers could reduce diagnostic uncertainty and improve identification of invasive bacterial infections.
Introduction
Children presenting at the emergency department (ED) still die from treatable invasive bacterial infections (IBI) due to delayed or missed diagnosis.1–3 For not missing one child with IBI, antibiotics are prescribed in children with self-limiting viral infections.4 The distinction between bacterial and viral infections based solely on clinical signs and symptoms is unreliable. Although C reactive protein (CRP) and procalcitonin are currently used as markers for bacterial infections, they measure non-specific inflammation and immunological responses. Recent studies focus on proteomic and transcriptomic approaches for finding new discriminators of bacterial and viral infections.5–8 Due to costs and limited resources, it is not feasible to apply new biomarkers to all febrile children. Therefore, prediction models are needed to identify risk groups where biomarkers can improve diagnosis.
Clinical prediction models that include clinical signs and CRP or procalcitonin have been developed to assist decision making in treatment of febrile children,9–15 and have focused on young infants to differentiate between patients at high risk or low risk for IBI (bacteraemia, meningitis, bone/joint infections). No clinical prediction models for IBI exists for older children who are also at risk for IBI.16 17 The Feverkidstool, developed for children aged <16 years, predicts risks for pneumonia and other serious bacterial infections which besides IBIs also includes bacterial infections of the urinary tract, gastrointestinal tract and soft tissue.
Although the Feverkidstool is extensively validated, the original population only included 21 IBI cases and important predictors for IBI such as non-blanching rash or neurological symptoms were not included. Several models yet exist for prediction of bacterial pneumonia and the impact of the original Feverkidstool on antibiotic use in respiratory tract infections is proven.18 Therefore, another model for bacterial pneumonia is not required. Furthermore, prediction of urinary tract infections may be less relevant as sensitive laboratory tests (urinalysis) are readily available for accurate diagnosis at ED visit. In addition, the Feverkidstool is developed in previous healthy children and is therefore not applicable for children with chronic conditions with higher risk of IBI. Hence, a new tool is required for early risk assessment of IBI in febrile children including all age ranges (0–18 years) and chronic conditions.
We aim (1) to derive and cross-validate a clinical prediction model including CRP to identify IBIs in febrile children presenting to different European EDs and (2) to identify patient groups which might benefit from new biomarkers.
Methods
Study design
This study is embedded in MOFICHE (Management and Outcome of Febrile children in Europe), an observational multicentre study, which is part of PERFORM (PErsonalized Risk assessment in Febrile illness to Optimise Real-life Management across the European Union) (www.perform2020.org).
Children aged from 0 to 18 years with temperature ≥38.0°C or fever <72 hours before ED visit were included. Twelve EDs participated in this study: Austria, Germany, Greece, Latvia, the Netherlands (n=3), Spain, Slovenia and the UK (n=3).19 Data were collected for at least 1 year from January 2017 to April 2018. Details of the study design have been described previously.20
For this study, we selected patients with CRP measurement and excluded patients with working diagnosis of urinary tract infections after first assessment at the ED.21 To identify IBI at the earliest opportunity, we included only the first ED visit for patients with IBI who repeatedly visited the ED within the same disease episode. Data were analysed according to a statistical analysis plan (online supplemental appendix 1).
Supplemental material
Collected data included age, sex, comorbidity (chronic condition expected to last ≥1 year),22 warning signs for identifying risk of serious illness (National Institute for Health and Care Excellence (NICE))23 (consciousness, ill appearance, work of breathing, meningeal signs, focal neurology, non-blanching rash, dehydration) and vital signs (heart rate, respiratory rate, oxygen saturation, temperature, capillary refill time). We collected CRP level (point-of-care or laboratory assay) and microbiologic cultures (blood, cerebrospinal fluid and other) ordered at the ED or at the first day of hospital admission on indication of the physician. Furthermore, we collected data of prescribed antibiotics and admission following ED visit.
Outcome
IBI included bacterial meningitis, bacteraemia and bacterial bone/joint infections, defined as culture or PCR detection of a single pathogenic bacterium in blood, cerebrospinal or synovial fluid. All cultures that were treated as contaminant and cultures growing contaminants were considered non-IBI (online supplemental appendix 2).24 Cultures growing a single contaminant or candida were defined positive in patients with malignancy, immunodeficiency, immunosuppressive drugs or a central catheter, since antimicrobial treatment is needed in these patients.
Model development
Descriptive and univariate logistic regression analyses were performed for children with and without IBI.
Sample size was estimated based on Riley et al.25 Assuming 16 predictors, a prevalence of 0.8% and an expected R2 of 0.0135 (15% of maximum achievable R2), a sample size of 10 587 with 85 cases would be sufficient. For model development,26 27 we considered predefined variables with predictive value for IBI: (1) variables in the Feverkidstool9 (age, sex, temperature, fever duration, tachypnoea and tachycardia defined by Advanced Paediatric Life Support,28 oxygen saturation <94%, capillary refill ≥3 s, work of breathing, ill appearance and CRP value), (2) NICE warnings signs (consciousness, meningeal signs, focal neurology, status epilepticus, non-blanching rash)23 and (3) complex chronic condition (≥2 body systems, malignancy or immunocompromised).22 Consciousness, meningeal signs and focal neurology were combined into a composite variable abnormal neurology. Linearity of continuous variables was assessed using restricted cubic splines. As in the Feverkidstool, age was modelled linear piecewise for children aged <1 year and >1 year and a logarithmic transformation for CRP was used. Outliers were truncated at the 0.01 percentile for temperature (35.7°C) and the 0.99 percentile for CRP (215 mg/L) and fever duration (8 days).
Variable selection was not influenced by the results of the univariate logistic regression analysis, but was performed using least absolute shrinkage and selection operator (LASSO), which reduces the degree of overfitting by shrinking large regression coefficients (detailed methods in online supplemental appendix 3).29 30 The final model was developed on data from all the 12 EDs. For the cross-validation, we created 5 ED groups; 1 group combined the data from the 8 EDs with <10 IBI cases and 4 groups were based on data from EDs with >10 IBI cases per ED: Slovenia, the Netherlands (n=2) and the UK (online supplemental appendix 4). Next, in cross-validation the model was repeatedly derived on four ED groups and validated on the fifth ED group, leading to five different cross-validations.31 The five cross-validations were pooled using a random-effects model. This cross-validation determines model performance most accurately and provides information on the heterogeneity of performance across different settings. This cross-validation is therefore superior to a single external validation.13 31 We assessed the discriminative ability by the area under the receiver operating curve (AUC), and calibration, the agreement between predicted risks and observed cases. We explored the impact of difference in case-mix heterogeneity on the discriminative ability of the model in the internal-external cross-validation. We used decision curve analysis to evaluate the net benefit of the prediction model.32 At different cut-offs for the individual probability of IBI according to the model, we assessed sensitivity, specificity, negative and positive likelihood ratios (LRs) . Missing values for the covariates were multiple imputed using the MICE package, resulting in 20 imputation sets (details in online supplemental appendix 3). Sensitivity analysis was performed in the population where missing CRP values were imputed. All analyses were performed in R V.3.6.
Results
Of 38 480 patients, 17 213 patients had CRP measurements. Patients with CRP measurements were more often ill-appearing and admitted than patients without CRP measurements (online supplemental appendix 5). We excluded 939 urinary tract infections and 6 repeated visits in the same disease period of patients with IBI, resulting in 16 268 patients. Of those, most common infections were the upper respiratory tract (45%), lower respiratory tract (18%), gastrointestinal tract (14%) and undifferentiated fever (9%). IBI was diagnosed in 135 patients (0.8%), and comprised 119 bacteraemias, 15 bacterial meningitis and 9 bone/joint infections (8 patients had concurrent infections). Main pathogens included Streptococcus pneumoniae (21%), Staphylococcus aureus (19%), Escherichia coli (10%), Neisseria meningitidis (7%) and coagulase-negative staphylococcus (7%) (figure 1, online supplemental appendix 6). Complex chronic conditions were present in 37% of patients with IBI vs 6% of patients without IBI. IBI incidence varied from 0.1% to 5.6% of patients per ED (online supplemental appendix 4).
Identified pathogens for invasive bacterial infections (n=135). CoNS, coagulase-negative staphylococci; spp, species.
Patients with IBI were similar in age and sex compared with patients without IBI. CRP level was higher in the IBI group (median 62 mg/L, IQR 21–144) than in the non-IBI group (median 16 mg/L, IQR 5–45) (p<0.01) (table 1). The majority of IBIs were treated with antibiotics (n=126, 93.3%) at first ED visit and all were treated with antibiotics in the disease course. The associations of the sole predictors with IBI are provided in online supplemental appendix 7.
Characteristics of patients with invasive bacterial infections and patients without invasive bacterial infections
The final model is presented in table 2. This model discriminated well (AUC 0.84 (95% CI 0.81 to 0.88)). In the cross-validation, the model discriminated moderate to well (range AUC 0.76–0.81) yielding a pooled AUC of 0.78 (95% CI 0.74 to 0.82) (figure 2). Calibration was poor to moderate for the different cross-validations (range slope: 0.45–0.81, range intercept −1.2 to 1.0) (online supplemental appendix 8). Apparent calibration was improved by adding an ED-specific variable for high (>2%) versus low (<2%) incidence of IBI (online supplemental appendix 9).
Discriminative value of the prediction model for invasive bacterial infection for five internal-external cross-validations. The model was repeatedly derived on four ED groups, and validated on the fifth ED group which was left out from the derivation. The five cross-validations were pooled using a random-effects model. More details are provided in figure A in online supplemental appendix 3. AUC, area under the receiver operating curve; ED, emergency department; NL, The Netherlands; UK, United Kingdom; UMC, University Medical Centre.
Model specification of multivariate logistic model for IBI
The diagnostic performance was good for the rule-out threshold of 0.1% with sensitivity of 0.97 (95% CI 0.93 to 0.99) and negative LR of 0.09 (95% CI 0.03 to 0.23) (table 3, online supplemental appendix 10). For the rule-in threshold of 2.0%, the model had specificity 0.94 (95% CI 0.94 to 0.95) and positive LR of 8.4 (95% CI 6.9 to 10.0). The intermediate thresholds of 0.1%–2.0% performed poorly (ranges: sensitivity 0.59–0.93, negative LR 0.14–0.47, specificity 0.52–0.88, positive LRs 1.9–4.8) and comprised 9784 (60.1%) patients. The rule-in threshold misclassified four patients with IBI from three different EDs, including two patients with arthritis, and two patients with a sinusitis and pneumonia resulting in bacteraemia. Three of these patients had CRP levels <10 mg/L and symptoms <1 day.
Diagnostic performance of the prediction model for different risk thresholds for invasive bacterial infection
In sensitivity analysis involving the population with imputed CRP levels (n=37 093, IBI n=135), model development yielded similar coefficients (online supplemental appendix 11).
Discussion
Based on the Feverkidstool and important predictors for early recognition of IBI, we derived and cross-validated a clinical prediction tool, in febrile children at different European EDs. The prediction model discriminated well between patients with and without IBI. The risk threshold of 0.1% has good rule-out value for IBI and thus decreases the risk of missing an IBI. The higher risk thresholds of >2.0% have good rule-in value and these thresholds can be used to identify patients at high risk of IBI to target treatment. The large number of patients with intermediate risk of 0.1%–2.0% for IBI is expected to benefit most from sensitive biomarkers.
Strengths of this study include the participation of 12 European EDs based in 8 countries with a broad population of febrile children of all ages and chronic conditions. Furthermore, we performed five cross-validations which provided us insight in heterogeneity between EDs, and improves the generalisability of our results. Second, we included a large number of IBI cases, while previous studies did not have sufficient cases to define a prediction model exclusively for IBI.9–11 Furthermore, our model involves accessible predictors as clinical symptoms and CRP level, which will facilitate implementation in practice. We provide clinical case examples of the model (online supplemental appendix 12) and, to help physicians to use this model in practice, a web-based digital calculator will be developed.
Our study has some limitations. First, we focused our study on patients who had CRP measurement on indication. This involved more severe illness than patients without CRP measurement. However, the CRP group reflect patients with diagnostic uncertainty and is more likely to benefit from a clinical prediction model. All patients with IBI had CRP measurement, leading to inclusion of all eligible IBIs in the main analysis. In our sensitivity analysis, predictors were similar in the model developed on imputed CRP levels. Therefore, model performance was not influenced by selection of patients with CRP measurement. Second, diagnostic tests were ordered according to usual care. If patients with an IBI did not have cultures taken >24 hours after hospital admission, this was not included in the data and these patients could have been misclassified as non-IBI. Since diagnostic workup is in general performed at the ED or <24 hours after presentation, this misclassification is minimised. Third, due to the low incidence of IBI, model performance was evaluated in cross-validation with a lower number of cases than is optimal for validation (100 cases).33 34Although discrimination of the model was good in the cross-validations, calibration was poor to moderate. The low incidence of IBI and other case-mix differences not taken into account by our model may have influenced model performance in the cross-validation. Our range of IBI incidence (range EDs 0.1%–5.6%) was comparable with IBI incidence in other studies including febrile population of all age ranges (range 0.4%–4.5%).9 11 35 Fourth, due to limited measurements of systolic blood pressure (14.7%) and procalcitonin in our cohort (1.6%), we were not able to include these as predictor. Lastly, data on individual immunisation status were not available and were not included in the model. In the clinical assessment of febrile patients, immunisation status should be taken into account.
Patients with and without IBI were discriminated well in the cross-validations. Calibration was poor to moderate indicating discrepancy between model predictions and the observed risk of IBI. Addition of the ED covariate of low/high incident IBI improved calibration, indicating that model performance is influenced by the likelihood of IBI in the ED. Therefore, ED incidence should be included in the model.
Clinical prediction models involving older children are the Feverkidstool and Irwin’s model, and predict pneumonia and other serious bacterial infections separately, whereas our model focuses on IBI. Discrimination of our model in cross-validation (pooled AUC: 0.78 (95% CI 0.74 to 0.82) was better compared with one external validation and similar to another external validation of the Feverkidstool for other serious bacterial infection.9 11 Unlike our study, these models were not based on an European-wide ED population. We recommend to use the Feverkidstool to guide antibiotic prescription in suspected lower respiratory tract infections18 and to use our model in febrile children to predict IBI. These two models, the original Feverkidstool and our model will be integrated in one electronic decision tool. For both implementation of the Feverkidstool and our model, measurement of (point-of-care) CRP is necessary. We do not recommend CRP measurement in all febrile children, but since CRP level is an important discriminator in bacterial and viral illness, measurement should be easily accessible to aid in the decision-making process at the ED.
Missing and undertreatment of IBI in children can lead to morbidity and mortality. Current practice is to start antibiotic treatment in patients at risk for bacterial infection awaiting culture results which take >48 hours. Since the low incidence of IBI, this leads to overuse of antibiotics and resources. The balance of not missing IBIs and overtreating self-limiting infections is delicate. Therefore, clinical prediction models can help in decision making at the ED. Our study showed that the low-risk threshold can be helpful to rule-out IBI and to reduce invasive diagnostics and antibiotic use.
Starting early treatment is key to prevent adverse outcomes due to IBI. The high risk threshold of >2.0% can be used for targeted treatment with intravenous antibiotics. Although our model was able to identify 38% of the study population as low or high risk, diagnostic uncertainty exist for the intermediate group (60%). In our study, this intermediate group with diagnostic uncertainty was estimated as 25% of the population of febrile children presenting to the ED, including patients without CRP measurement. Additional diagnostics including procalcitonin, repeated CRP measurement36 or novel sensitive biomarkers may be helpful in the decision making for this intermediate risk group. The potential benefit of additional diagnostics using these risk thresholds will need to be evaluated in future studies.
Conclusion
Based on the Feverkidstool and important clinical predictors, we derived and cross-validated a clinical prediction model for early detection of IBI in febrile children in an European-wide cohort. Where the rule-in threshold of this model could target early treatment to reduce adverse outcomes from IBI, the rule-out threshold has the potential to reduce unnecessary use of invasive diagnostics and antibiotics. However, more than half of the population was at intermediate risk. In this group, sensitive, new biomarkers could improve identification of IBI and could potentially reduce unnecessary antibiotic use.
Data availability statement
An anonymized data set containing individual participant data is available in a public data repository: https://data.hpc.imperial.ac.uk/resolve/?doi=7549. DOI: 10.14469/hpc/7549. For inquiries to obtain the full dataset, please contact the data manager of the PERFORM consortium (Tisham.de08@imperial.ac. uk).
Ethics statements
Ethics approval
The study was approved by all the participating hospitals. No informed consent was needed for this study. Austria (Ethikkommission Medizinische Universitat Graz, ID: 28-518 ex 15/16), Germany (Ethikkommission Bei Der LMU München, ID: 699-16), Greece (Ethics committee, ID: 9683/18.07.2016), Latvia (Centrala medicinas etikas komiteja, ID: 14.07.201 6. No. Il 16-07-14), Slovenia (Republic of Slovenia National Medical Ethics Committee, ID: ID: 0120-483/2016-3), Spain (Comité Autonómico de Ética de la Investigación de Galicia, ID: 2016/331), The Netherlands (Commissie Mensgebonden onderzoek, ID: NL58103.091.16), UK (Ethics Committee, ID: 16/LO/1684, IRAS application no. 209035, confidentiality advisory group reference: 16/CAG/0136).
Acknowledgments
The authors would like to thank the members of the PERFORM consortium as listed in online supplemental appendix 13.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @rgnijman, @CarrolEnitan
Correction notice This paper has been amended since it was published online. The data sharing statement has been updated.
Contributors Conceptualisation: NNH, DB, RGN, DN, JAH, AB, UvB, EC, IE, ME, MvdF, RdG, BK, EL, IM, FM-T, MP, FS, MT, DZ, WZ, ML, CV, HAM. Data curation: NNH, DB, RGN, JAH, AB, UvB, EC, IE, ME, MvdF, RdG, JAH, BK, EL, IM, FM-T, MP, FS, MT, DZ, WZ, ML, CV, HAM. Formal analysis: NNH, DN. Methodology: NNH, DB, RGN, DN, JAH, AB, UvB, EC, IE, ME, MvdF, RdG, BK, EL, FM-T, DN, MP, MT, DZ, CV, WZ, ML, HAM. Supervision: CV, HAM. Writing—original draft: NNH. Writing—review and editing: NNH, DB, RN, DN, JAH, AB, UvB, EC, IE, ME, MvdF, RdG, BK, EL, IM, FM-T, MP, FS, MT, DZ, WZ, ML, CV, HAM.
Funding This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 668303. The research was supported by the National Institute for Health Research Biomedical Research Centres at Imperial College London, Newcastle Hospitals NHS Foundation Trust and Newcastle University.
Disclaimer The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
Competing interests None declared.
Patient and public involvement statement The design of the study did not involve patients. The study results will be disseminated to members of the public, patients and experts through social media and networks.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.