Article Text
PDF
Abstract
Background Despite fewer serious infections presenting to the children's emergency department (ED), hospital admissions of children with febrile illness have increased. We review evidence for the use of decision rules to increase the safe discharge of these children from the ED.
Methods A systematic review of prospective studies of decision rules for the discharge of children with febrile illness, and prediction rules for the diagnosis of serious infections in children presenting to ED. We reviewed the MEDLINE database, Cochrane Library and hand searched the bibliographies of related studies. The search was limited to the English language.
Results Thirty-three studies were identified. Fourteen reported low-risk criteria to rule out serious bacterial infection (SBI) in infants less than 3 months of age. In this group, clinical tools such as the Rochester and Philadelphia criteria support the safe discharge of low-risk infants without empirical antibiotics. Seventeen studies reported prediction rules in older children, though only four included children over 3 years. Two impact studies based upon multivariable prediction models failed to demonstrate any impact on rates of discharge from ED.
Conclusions The use of clinical prediction models can improve discrimination between serious and self-limiting infections in children. The application of low-risk thresholds may help to rule out serious infections and discharge children from the ED without empirical antibiotics. A growing evidence base for prediction rules has so far failed to translate into validated rules to aid decision-making. Future work should evaluate decision rules in well designed impact studies, focusing on the need for hospital admission and antibiotic therapy.
- Accident & Emergency
- Infectious Diseases
- General Paediatrics
- Health Service
Statistics from Altmetric.com
What is already known on this topic
In the UK, children with febrile illness are increasingly likely to be admitted to hospital. Many admissions are short, uncomplicated admissions with self-limiting infections.
The National Institute for Health and Care Excellence guideline ‘Feverish illness in children’ on the management of children with febrile illness has limited capacity to discriminate between serious and self-limiting infections.
Risk prediction models combining clinical and biomarker variables have the potential to identify low-risk children in whom serious infection can be ruled out.
What this study adds
Summary estimates of sensitivity and negative likelihood ratio reveal that both Rochester and Philadelphia criteria support clinicians to rule out serious illness in young infants with febrile illness.
Despite encouraging estimates of diagnostic accuracy from multivariable risk prediction models, no impact on clinical decision-making has yet been demonstrated in this context.
Future impact studies should focus on the need for admission and antibiotic therapy, and investigate the behaviours which influence decisions to admit or discharge children.
Background
Febrile illnesses in children are a common presentation to the emergency department (ED).1 In the UK, despite reductions in the incidence of pneumonia, meningitis and septicaemia,2–4 hospital admissions of children with febrile illness are increasing. Admissions of children with acute infections increased by 30% between 1999 and 2010. Uncomplicated, short-stay admissions of children with ‘primary-care-sensitive’ infections—considered appropriately managed in the community—doubled over this period.5 ,6
Understanding this increase in uncomplicated, short-stay admissions is a priority. Between 2007–2008 and 2013–2014, the number of children attending EDs increased from 3.5 million to 4.9 million.7 ,8 In all age groups, increased attendance to major EDs is associated with an increased proportion of attendances resulting in admission.9 Further, the introduction of 4 h ED waiting time targets may have prompted decisions to admit rather than observe children. Together with changes to medical training and working patterns,10 these factors may have encouraged a minimum-risk approach to children with febrile illnesses and the hospitalisation of increasing numbers with self-limiting infections.
Evaluating risk of serious illness in children with fever is challenging. In the UK, clinicians are supported by the National Institute for Health and Care Excellence (NICE) guideline ‘Feverish illness in children’. This is a comprehensive guideline designed to support clinicians in primary and secondary care in the assessment and management of children with fever. It includes a ‘traffic light’-based system to identify clinical signs which predict risk of serious illness. Various studies have evaluated risk prediction rules for serious illness in children with fever and these have recently been reviewed.11 Such rules allow the stratification of risk, and the identification of low-risk thresholds may help to rule out serious illness, an approach undertaken in young infants.12 Despite this opportunity, little attention has focused on how such risk prediction influences decision-making relating to admission and discharge.
We systematically reviewed impact studies of decision rules to increase the safe discharge of children with febrile illness of all ages from the ED. We also reviewed diagnostic accuracy studies of prediction rules that identify children at low risk of serious infection with the potential to impact rates of discharge.
Methods
The population of interest is children of all ages presenting to the ED with an acute febrile illness. For impact studies, we included prospective studies of decision rules (all study designs), for which outcomes included rate of discharge from the ED. For diagnostic accuracy studies, prospective studies of children with febrile illness in which the target condition was serious illness, serious infection or serious bacterial infection (SBI) were included. Studies investigating restricted clinical presentations (such as meningitis) were excluded on the basis that only in ruling out all serious infection is a clinician able to discharge the child from the ED. Index tests were prediction rules composed of clinical or biomarker variables. Individual biomarkers have been extensively evaluated in the diagnosis of serious infection, and recently reviewed.13 No single biomarker is able to reliably rule out serious infection, and so we excluded studies of individual biomarkers, but included studies of multiple biomarkers, or biomarkers in combination with clinical signs. Similarly, studies of individual adjunctive tests, such as respiratory viral assays, were excluded. Reference standards were microbiological confirmation of serious infection, or consensus opinion based on predefined criteria.
A literature search of the MEDLINE database (1950–) was undertaken in October 2014 using the search strategy in box 1, limited to the English language. The search was updated and extended to include a search of the Cochrane database of systematic reviews in August 2015.
Search strategy
(“discharge”[MeSH Terms] OR “discharge”[All Fields]) OR (“diagnosis”[MeSH Terms] OR “diagnosis”[All Fields] OR “diagnostic”[MeSH Terms] OR “diagnostic”[All Fields] OR “predict*”[MeSH Terms] OR “predict*”[All Fields]) AND (“infection”[MeSH Terms] OR “infection”[All Fields]) AND (“emergency”[MeSH terms] OR “emergency”[All Fields]) AND (“infant”[MeSH Terms] OR “child”[MeSH Terms] OR “adolescent”[MeSH Terms]).
Returned titles were independently inspected by two authors (ADI and JW). The full text manuscripts of relevant abstracts were obtained and prospective studies included. Hand searching of the bibliographies of retrieved articles was undertaken.
Diagnostic accuracy studies were independently evaluated for quality by ADI and JW using the QUADAS-2 framework,14 while impact studies were evaluated for risk of bias using the tool developed by the Cochrane collaboration.15 Disagreement was resolved by discussion. Trial characteristics and results were extracted into Excel forms by ADI. For the objective of ruling out serious infection, negative likelihood ratios (NLRs) were calculated by extracting the number of true and false positives and negatives and creating 2×2 tables in the ‘epiR’ package of R.16 ,17 No attempt was made to contact study authors if data were insufficient to calculate likelihood ratios.
Studies reporting the same index test were combined using the bivariate random effects model of Reitsma in the ‘mada’ R package.18 Summary estimates of NLRs were calculated using the SummaryPts() function as described by Zwinderman and Bossuyt.19 The effect of the specific index test used was estimated by comparing bivariate meta-regression models using a maximum likelihood method.18
Results
Thirty-three primary studies were identified. Figure 1 provides a flow diagram of the search process. Two impact studies evaluated decision rules,20 ,21 and the remainder were diagnostic accuracy studies of prediction rules. The characteristics of included studies are available as online supplementary tables. There was a well established literature on prediction rules to identify low-risk young infants (less than 3 months of age). As this mirrors the clinical situation, with young infants being evaluated differently than older children, we report literature relating to young infants and older children separately.
PRISMA flow diagram.22
Quality of the diagnostic accuracy studies was moderate. The main risk of bias related to the use of suitable reference standards for the diagnosis of serious infection or SBI. Few studies specified whether subjects were classified blinded to the index test, and a number were unclear about the nature of follow-up used to ensure subjects were not misclassified. The applicability of primary studies was frequently limited by the selection of children according to age and temperature on presentation (figure 2). Both impact studies were considered at high risk of bias, owing to the inability to blind participants and investigators. The quality of individual diagnostic accuracy studies and risk of bias in impact studies are detailed in the online supplementary appendix.
Risk of bias, and concerns relating to applicability of primary studies of young infants (top), and older children (bottom). See online supplementary appendix for details.
There was substantial clinical diversity between the studies. In addition, resulting from the selection criteria used, the proportion of infants considered low risk varied substantially, as did the proportion with serious outcomes. There were a large number of different prediction rules (index tests) combining clinical and biomarker variables. Few had undergone external validation.
Identification of low-risk young infants with febrile illness in the ED
Fourteen prospective studies examined low-risk criteria to rule out serious infection in healthy infants less than 3 months of age. The proportion of infants with serious outcomes ranged from 4.5% to 29.4%, while the proportion of infants considered low risk ranged from 19% to 72%.
A number of risk scores were identified. Two (the ‘Rochester’ and ‘Philadelphia’ criteria, detailed in table 1) have undergone external validation in prospective studies. Forest plots of the sensitivity and specificity of these are shown in figure 3. Few low-risk infants were misclassified by the Rochester and Philadelphia criteria (30/3928 and 2/1350, respectively) and no adverse outcomes identified. Pooled estimates of sensitivity and NLR are shown in table 2. Individual study results and summary estimates are plotted as a summary receiver operator characteristic (ROC) curve (figure 4). Inclusion of the type of index test in a meta-regression model improved the fit of the model (χ2 9.45, df=2, p<0.01). The use of the Philadelphia criteria was associated with a reduction in specificity (p=0.004), compared with the Rochester criteria, with no improvement in sensitivity (p=0.16).
Clinical features and laboratory tests included in risk prediction scores in young infants
Pooled estimates of sensitivity and negative likelihood ratio (NLR) of the Rochester and Philadelphia criteria
Forest plot of sensitivity and specificity of the Rochester (top) and Philadelphia (bottom) criteria. Note the different scales.
Combined summary receiver operator characteristic (ROC) curve for studies of Rochester and Philadelphia criteria showing point and summary estimates with confidence regions.
In those studies in which low-risk infants were discharged home, no adverse outcomes were identified. In three studies which discharged infants without antibiotics, no cases of SBI were observed. A small number of infants were admitted, but none found to have serious illness.24–26 Two studies which discharged low-risk infants with ambulatory ceftriaxone each identified a single case of missed bacteraemia. The first study (n=86) used the Rochester criteria, and the organism identified was Neisseria meningitidis. In the second, the risk score used was the ‘Milwaukee protocol’ and the organism was Moraxella catarrhalis. Both infants were well at follow-up.27 ,28 A large observational study of low-risk criteria from Boston using ambulatory ceftriaxone revealed a surprisingly high rate of SBI (27/503). This included eight infants with bacteraemia, though no adverse outcomes occurred. No subsequent evaluations have been undertaken.29
Ruling out SBI in children with febrile illness older than 3 months presenting to the ED
Seventeen prospective diagnostic accuracy studies reported prediction rules for SBI in older children with febrile illness presenting to the ED. Significant clinical diversity was evident across the studies. The proportion of children with serious outcomes ranged from 3.8% to 44.7%.
The earliest risk prediction score to be derived in children with febrile illness was the Yale Observation Scale (YOS).30 We identified five external validations in this age group.31–35 One was undertaken in infants aged 2–6 months, the remainder in children up to 3 years. Summary estimates of sensitivity and NLR were 48.5% (22.6% to 75.1%) and 0.67 (0.34 to 0.98), respectively.
Two large prospective studies evaluated the predictive value of signs included in the ‘traffic light system’ of the NICE guideline in the children's ED.36 ,37 In each, the absence of ‘amber’ or ‘red’ signs had limited value in ruling out SBI (NLR 0.5 and 0.77, respectively), while their presence failed to rule in SBI. De et al37 reported that the absence of ‘amber’ or ‘red’ signs failed particularly to rule out urinary tract infection (UTI). However, the addition of a positive urinalysis (as advocated by the guideline) improved the sensitivity of detection of UTI from 78.5% to 93.9%. A separate evaluation of ‘red’ features in more than 6000 children reinforced that the absence of red flag signs fails to rule out serious infection.38
The Manchester Triage System (MTS) was evaluated in two diagnostic accuracy studies of children with febrile illness. Thompson et al36 suggested that the two most urgent MTS categories were of value in ruling in ‘serious or intermediate’ infection (positive likelihood ratio 3.8), though this was not reproduced in a study by Nijman et al.39 A low-risk evaluation failed to rule out serious infection in either study (NLR 0.8 and 0.84, respectively).
A number of studies derived prediction rules for SBI in children. Craig et al40 used multinomial regression to derive risk prediction models for each diagnosis of UTI, pneumonia and bacteraemia in more than 15 000 children aged under 5 years. The models demonstrated reasonable discrimination in a validation sample of 5584 children (c statistic 0.78, 0.84 and 0.74 for the diagnosis of UTI, pneumonia and bacteraemia, respectively).
Nijman et al derived models for pneumonia and ‘other SBIs’ in children under 15 years. The models discriminated well (c statistic 0.81 and 0.86 for pneumonia and other SBIs, respectively), and were well calibrated. Adding C-reactive protein (CRP) to a clinical model improved discrimination. A low-risk threshold was valuable in ruling out pneumonia (NLR 0.12), though less good at ruling out ‘other SBIs’ (NLR 0.59).41
Two further scores have undergone external validation. Bleeker et al42 combined clinical and laboratory variables (white cell count (WCC), CRP and urinalysis) in children under 3 years. A low-risk score achieved a NLR of 0.19 in the derivation study but failed to discriminate well on external validation.43 The ‘Lab score’ combined CRP, procalcitonin (PCT) and urinalysis.44 Performance characteristics from three studies of the Lab score are illustrated in figure 5. Pooled sensitivity was estimated to be 82.3% (55.7% to 94.5%) and NLR 0.23 (0.07 to 0.52).
Forest plot of sensitivity and specificity of primary studies of the Lab score.
Impact studies of decision rules
Two impact studies evaluated decision rules in the management of children with febrile illness in the ED.20 ,21 One rule was based upon the Lab score and undertaken in children under 3 years (n=271). The other was based on the risk model of Nijman et al (n=439). Neither study demonstrated an increase in rates of ED discharge (Lab score 87/131 vs 90/140, p=0.8, and Nijman model 193/219 vs 197/220, p=0.7). No impact was seen on rates of antibiotic prescribing.
Discussion
Main findings
We set out to identify studies which support the objective of safely discharging children with febrile illness from the ED. We identified two impact studies of decision rules. There is presently no evidence that these rules influence decision-making with regard to discharging children home or withholding antibiotics.
We identified a number of diagnostic accuracy studies which support the use of low-risk criteria to rule out serious infection in infants less than 3 months of age. Using both the Rochester and Philadelphia criteria, when adequate community support is available, it is possible to safely discharge low-risk young infants from the ED without antibiotics. Our analysis suggests that the Rochester criteria achieve the same sensitivity as the Philadelphia criteria with greater specificity. The need for ambulatory antibiotics in this setting is unclear. Risk models that predict safe discharge without antibiotics are preferable to routine antibiotic use. A case of meningococcal bacteraemia discharged home with ceftriaxone serves as a cautionary tale, however. Those studies which used ambulatory antibiotics identified a greater number of ‘missed’ SBI.12 It appears that clinicians may adopt a higher risk threshold when reassured by the use of antibiotics.
In children over 3 months with febrile illness, a number of robust studies now report reliable risk prediction models combining clinical variables with adjunctive tests (such as CRP and PCT). External validations of the Lab score and the Nijman risk model are encouraging. By contrast, the NICE guideline ‘Feverish illness in children’ demonstrates limited capacity to discriminate between serious and self-limiting infection.36 ,37
Comparison with other studies
We identified three reviews of diagnostic accuracy studies in children with febrile illness.11 ,12 ,45 Two summarised the use of low-risk criteria in infants less than 3 months of age. Our pooled estimates of the sensitivity of the Rochester and Philadelphia criteria are similar to those of Hui et al45 which included retrospective studies (94% and 93%, respectively). Though summary estimates of sensitivity and NLR were not provided in the review by Huppler et al12 (which pooled relative risks of SBI in low-risk and high-risk infants), its conclusion that low-risk criteria are sufficiently sensitive to rule out SBI is consistent with our own. A large retrospective study of the Lab score in 1012 young infants with febrile illness suggested that, though the score appeared valuable at ruling in SBI, it was unable to rule out SBI (NLR 0.5 for SBI, and 0.85 for bacteraemia or meningitis).46
In older children, a review by Thompson et al11 externally validated published prediction rules using other published datasets. Our summary estimate of NLR for YOS is consistent with estimates in the review (range 0.58–1.01), and affirms that the YOS is inappropriate for the purpose of ruling out serious infection. The review also highlighted a large multicentre study of children in primary care which derived a classification tree supporting clinicians to rule out SBI (NLR 0.04, 95% CI 0.01 to 0.2). External validation in ED datasets was disappointing, however.11
One study formally evaluated the use of respiratory virus tests in the context of a clinical prediction rule.47 In multivariable analysis a positive viral identification reduced the likelihood of SBI in young infants with febrile illness. A number of prospective studies corroborated this finding in young infants with respiratory syncytial virus48 and influenza,49 ,50 though the presence of virus alone was insufficient to rule out SBI. The impact of respiratory viral testing was summarised in a Cochrane review of randomised controlled trials including more than 1500 children. The review concluded that there was insufficient evidence of a reduction in antibiotic use or hospital admission as a result of the use of the test.51
Limitations of the evidence
The quality of the diagnostic accuracy studies reported was moderate, with particular concerns regarding patient selection and external applicability. As no gold standard reference test for the diagnosis of SBI or ‘serious infection’ exists, some misclassification of children with febrile illness is likely.
Limitations of the review
The review is limited to studies in the English language and all of the evidence presented is from high-resource settings.
We hypothesise that short-stay admissions of children with febrile illness to hospital occur primarily as a result of the diagnostic uncertainty faced by clinicians in the ED. We recognise, however, that decisions to admit children also relate to complications such as hypoxia and dehydration. Low-risk criteria applied to all children with febrile illness in the ED would need to incorporate such clinical signs. The decision to discharge a child also incorporates subjective factors, including a consideration of the family. It will be necessary to bring together clinician and parent perspectives to define ‘safe discharge’ and to understand the most effective means of safety netting.52
Next steps
The value of reliable risk prediction can only be realised by demonstrating improvements in decision-making in a well designed impact study.53 This is an overlooked aspect of diagnostic research. In children, very few prediction rules have reached this stage.54
An appropriate decision rule will require the establishment of acceptable thresholds of risk, for which it will be necessary to engage clinicians and parents. The model of Nijman et al which achieved good discrimination on external validation yields a continuous numeric likelihood of SBI. Translating risk thresholds (eg,‘low, <5%’, ‘intermediate, 5–20%’, ‘high, >20%’) into treatment decisions is the next stage if this approach is to have an impact on the ED. However, treatment decisions are complex clinical behaviours and understanding the behaviours influencing treatment decisions will be an important aspect of successful implementation. This has been an essential component of recent efforts to improve rational antimicrobial use.55
We propose an alternative to the use of SBI as an outcome of interest in diagnostic accuracy studies. As demonstrated in this and other reviews, the lack of a suitable reference standard introduces the possibility of misclassification and bias.56 Furthermore, its identification fails to relate to measurable clinical outcomes of interest, such as the need for hospital admission or antibiotic administration. Some serious infections require admission (gastroenteritis and dehydration, bronchiolitis needing respiratory support), but not antibiotics. Many SBIs conversely do not require admission. Future diagnostic accuracy and impact studies should evaluate risk of outcomes like admission, and requirement for empirical therapy. We propose an alternative classification of outcomes around which decision rules could be effectively evaluated (figure 6).
Proposed outcome categories for diagnostic accuracy and impact studies in children with febrile illness in the emergency department. SBI, serious bacterial infection; UTI, urinary tract infection.
Conclusions
Risk prediction models to identify low-risk children with febrile illness in the ED have the potential to reduce unnecessary hospital admission. Though substantial evidence supports their use in this context, no studies have yet demonstrated the impact on decision-making in clinical practice. There is a clear need to understand the behaviours influencing decisions to admit or discharge children with febrile illness and for well designed impact studies of decision rules to guide their management.
Acknowledgments
We gratefully acknowledge Dr Yingfen Hsia for helpful comments on statistical methods.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Twitter Follow Adam Irwin at @adamdirwin
Contributors ADI conceived the study, performed the search and evaluated the evidence, performed statistical analysis, and drafted the original manuscript. JW performed the search and evaluated the evidence and reviewed and revised the manuscript. KLD reviewed and revised the manuscript. SL reviewed and revised the manuscript. MS conceived the study, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.