Article Text

Cohort study to test the predictability of the NHS Institute for Innovation and Improvement Paediatric Early Warning System
  1. B W Mason1,
  2. E D Edwards2,
  3. A Oliver3,
  4. C V E Powell4
  1. 1Public Health Wales, Cardiff, UK
  2. 2Department of Paediatrics, Morriston Hospital, Swansea, UK
  3. 3Department of Paediatric Intensive Care, Children's Hospital for Wales, Cardiff, UK
  4. 4Department of Child Health, Children's Hospital for Wales, Institute of Molecular and Experimental Medicine, School of Medicine, Cardiff University, Cardiff, UK
  1. Correspondence to Dr E D Edwards, Department of Paediatrics, Morriston hospital, Swansea, SA6 6NL, UK; dawn.edwards{at}wales.nhs.uk

Abstract

Objective To test the predictability of the National Health Service Institute for Innovation and Improvement (NHSIII) Paediatric Early Warning System (PEWS) score to identify children at risk of developing critical illness.

Design Cohort study.

Setting Admissions to all paediatric wards at the University Hospital of Wales between 1 December 2005 and 30 November 2006.

Outcome measures Unscheduled paediatric high dependency unit (PHDU) admission, paediatric intensive care unit (PICU) admission and death.

Results There were 9075 clinical observations from 1000 children. An NHSIII PEWS score of 2 or more, which triggers review, has a sensitivity of 73.2% (95% CI 62.2% to 82.4%), specificity of 75.2% (95% CI 74.3% to 76.1%), positive predictive value (PPV) of 2.6% (95% CI 2.0% to 3.4%), negative predictive value of 99.7% (95% CI 99.5% to 99.8%) and positive likelihood ratio of 3.0 (95% CI 2.6 to 3.4) for predicting PHDU admission, PICU admission or death. Six (37.5%) of the 16 children with an adverse outcome did not have an abnormal NHSIII PEWS score. The area under the receiver operating characteristic curve for the NHSIII PEWS score was 0.83 (95% CI 0.77 to 0.88).

Conclusions The NHSIII PEWS has a low PPV and its full implementation would result in a large number of false positive triggers. The issue with PEWS scores or triggers is neither their sensitivity nor children with high scores which require clinical interventions who are not ‘false positives’; but their low specificity and low PPV arising from the large number of children with low but raised scores.

  • Health services research
  • General Paediatrics

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

What is already known on this topic

  • It has been recommended that Paediatric Early Warning Systems (PEWS) are implemented in UK hospitals, despite the absence of definitive evidence of effectiveness.

  • The performance characteristics of activation criteria for PEWS have not been fully established.

  • Activation criteria are part of a complex intervention designed to reduce paediatric mortality; this complex intervention has not been clearly defined or consistently implemented.

What this study adds

  • The National Health Service Institute for Innovation and Improvement (NHSIII) PEWS had reasonable sensitivity, but at the cost of low specificity and positive predictive value.

  • The performance characteristics of the NHSII PEWS activation criteria are similar to other systems.

  • The low specificity and low positive predictive value of the NHSIII PEWS score would generate a significant workload for clinical staff if it was fully implemented as designed.

Introduction

Since 2005 the proportion of hospitals in Great Britain using a Paediatric Early Warning System (PEWS) has increased from 22% to 85%.1 The National Health Service Institute for Innovation and Improvement (NHSIII) PEWS is the second most frequently used.1 The NHSIII PEWS can be introduced in a district general hospital as it does not require specialist staff for a medical emergency or rapid response team, but instead uses existing staff to respond to abnormal scores. We report the first published data which validate the performance characteristics of the NHSIII PEWS score.

The NHSIII PEWS has four age-specific charts which use six criteria to calculate the PEWS score (see online supplementary appendix 1).2 A higher score triggers a response from more senior staff.

Method

Data collection

Data were collected to validate another PEWS.3 In summary, paediatric (age 0–16 years) inpatient admissions, excluding direct admissions to the paediatric intensive care unit (PICU) or the paediatric high dependency unit (PHDU), were included in the study. The frequency of observations was determined by the clinical care policy at that time. Staff did not calculate a PEWS score and response to the observations was determined by normal clinical care. Adverse outcomes were defined as PHDU admission, PICU admission and death.

Date extraction

Data were available to provide a measure of all six components of the NHSIII score. Three components (heart rate, respiratory rate and receipt of oxygen) are objectively defined in the NHSIII PEWS and data were available to measure them precisely. Two components (respiratory distress and conscious level) are not objectively defined in the NHSIII PEWS. Positive score for these NHSIII PEWS components was determined by signs of respiratory distress as described in the Advanced Paediatric Life Support guidelines (Stata Statistical Software release 11.2. Texas: Stata Corporation, 2011); and, if the child was only responding to voice or less. Nurse or doctor worried was prospectively recorded in our original study; information on family concern was not. The NHSIII PEWS component ‘Doctor/Nurse/Family concern?’ was positive if there were doctor or nurse concerns.

Data analysis

An NHSIII PEWS score was calculated for each set of observations collected at a single point in time between admission and the occurrence of an adverse outcome or discharge. A set of observations was composed of the six component observations used to calculate the NHSIII PEWS score. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), likelihood ratio positive test and likelihood ratio negative test were calculated for each NHSIII PEWS score using MedCalc.4 Receiver operating characteristic (ROC) analysis was performed using Stata V.11.2 to study the ability of the NHSIII score to discriminate between children who went on to develop an adverse outcome and those who did not. The analysis was repeated for each of the four age ranges defined by the NHSIII PEWS.

For the purpose of the analysis any missing data were assumed to be normal. The analysis was repeated, limiting the data to NHSIII PEWS scores that were calculated from complete data when all six components of the score were recorded at a point in time.

Results

Performance characteristics of the NHSIII PEWS score

Data were available on 1000 patients on whom 9075 sets of observations were performed at a point in time. Sixteen children had an adverse outcome, 13 were admitted to PHDU, four of these subsequently transferred to PICU and three were admitted from the ward to PICU. There were no deaths. Six of these 16 children (37.5%) did not have an NHSIII PEWS score that would have triggered review (score of 2 or greater) prior to the adverse outcome. Fifty-eight per cent of children who did not have an adverse outcome would have been reviewed at least once during the admission (table 1). In total, 60 reviews would have been triggered in children with an adverse outcome and 2230 reviews in children without an adverse outcome (table 2).

Table 1

Number of sets of observations during whole admission with a NHSIII PEWS score that would have triggered review of child (score ≥2), in children with or without an adverse outcome

Table 2

Number of sets of observations for each NHSIII PEWS score, in children with or without an adverse outcome

An NHSIII PEWS score of 2 or more, which triggers a review, has a sensitivity of 73.2%, specificity of 75.2%, PPV of 2.6%, NPV of 99.7% and a positive likelihood ratio of 3.0 for predicting an adverse outcome. ROC analysis demonstrated an area under the curve (AUC) of 0.83 (table 4). The score cut-off that maximised the sum of sensitivity and specificity was 3, the score that triggers review by a junior doctor (table 3). The sensitivity of the NHSIII PEWS score was lowest and specificity highest in children aged 0–11 months (table 4).

Table 3

Performance characteristics of each NHSIII PEWS score

Table 4

Performance characteristics of NHSIII PEWS score ≥2

The criterion used to calculate the NHSIII PEWS score was incomplete, with 87% heart rate, 79% respiratory rate, 76% receipt of oxygen, 19% respiratory distress, 37% conscious level and 21% doctor/nurse worried recorded. All six criteria were completely recorded in 5% (479/9075) of the sets of observations, the performance characteristics of the NHSIII PEWS improved when analysis was restricted to these observations (table 4).

Discussion

A PEWS is a complex intervention combining education, a score or activation criterion and a response mechanism to assess and treat the child. PEWS aim to identify children at risk of deterioration, to trigger assessment and early intervention to reduce death or serious morbidity. The performance characteristics of the NHSIII PEWS score are similar to that of other published scores used on inpatients3 ,5–7 and in an emergency department.8 Reported sensitivity in all settings ranges from 61.3 to 94.4, specificity 25.2 to 95.0 and AUC from 0.60 to 0.90.3 ,5–8 None of the PEWS scores had both high sensitivity and specificity. The sensitivity of the NHSIII PEWS score means it would only detect around three quarters of children who developed critical illness. In common with other scores, the low specificity means the majority of children have an abnormal score during an admission and the majority of triggers are false positives. An NHSIII PEWS score of 3 maximises the sum of sensitivity and specificity and may be a more appropriate threshold for triggering review.

The study had a number of limitations. Data were incomplete because observations were not performed or recorded.9 If missing data were abnormal and not normal as assumed, the specificity and the PPV are likely to have been lower than measured. Analysis of the complete data, when all six components were recorded, led to a modest and non-significant improvement in the AUC. Data on ‘family concerned’ were not available, its inclusion might have improved the sensitivity of the tool but further reduce both specificity and PPV.

Six children who did not have an abnormal NHSIII PEWS score prior to an adverse outcome were admitted to HDU or PICU following a single set of observations. One child with bronchiolitis was admitted to PICU and five children were admitted to HDU with a diagnosis of bronchiolitis, convulsions, croup, meningococcal septicaemia or viral illness. The normal NHSIII PEWS score in these six children may have arisen from either incomplete measurement or recording of observations. The child with a NHSIII PEWS score of 6 (abnormal on all 6 parameters) who did not have an adverse outcome had a lower respiratory tract infection. The child had a series of 31 sets of observations; these were initially abnormal but stabilised. The issue with PEWS scores is not their sensitivity or children with high scores which require additional intervention; but their low specificity and low PPV arising from the large number of children with low but raised scores.

To address the low specificity some units use the escalation criteria only if the PEWS remains elevated after simple measure like undressing a feverish child or calming a crying infant. Such approaches should be an intrinsic part of the design of a PEWS. The measures, when they should be used and the timing of repeat observations should be documented as part of the tool.

No paediatric randomised control trials of the effectiveness of a PEWS have been published to date and results from before and after studies are inconsistent.10–13 These studies used a PEWS to trigger assessment by a team which included specialist staff from PICU, who are only available in tertiary hospitals. The NHSIII PEWS can be introduced in district general hospitals because it uses existing staff to provide the response to abnormal scores. No randomised controlled trials or quasi experimental studies investigating the effectiveness of a PEWS without a specialist response team have been published. A PEWS has the potential to save lives but the evidence base is still lacking.

A PEWS is part of a wider system of clinical care. Further research is needed to: identify through a systematic literature review the evidence for the core components of a PEWS score; develop a PEWS implementation package for prospective evaluation; evaluate the ability of the PEWS score to identify serious illness and reduce clinical events; identify contextual factors that are consequential for PEWS effectiveness and identify key ingredients of successful implementation and normalisation.14

The main technical limitation of the NHSIII PEWS, and others systems, is low specificity and PPV. The performance characteristics of the NHSIII PEWS and its effect on patient outcomes should be measured in a prospective implementation study.

References

View Abstract

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors All authors contributed to the conception of the work; EDE and AO to the acquisition of data, BWM to the analysis of data, CVEP to the interpretation of data for the work. BWM and EDE drafted the manuscript; CVEP and AO revised it critically for important intellectual content. All authors approved the final version of the manuscript submitted for publication.

  • Competing interests None declared.

  • Ethics approval The original study3 was approved by the Trust Research and Development Committee and ethical approval was granted by the South East Wales Local Research Ethics Committee. All patient identifiers were removed from the dataset at the time of analysis of our original study, and this study was a reanalysis of a totally anonymous dataset.

  • Provenance and peer review Not commissioned; externally peer reviewed.