Jump to comment:

• Response from the authors
  Susan M Chapman, Jo Wray, Kate Oulton, Samiran Ray and Mark J Peters
  Published on: 10 August 2017
• Content vs Context in scoring systems
  Damian T Roland
  Published on: 3 August 2017
• "Human factors matter" - Statistical analysis of performance of trigger systems misses the point
  Ronny Cheung, Peter Lachman
  Published on: 19 June 2017

• Published on: 10 August 2017

  Response from the authors

  • Susan M Chapman, Gulf Regional Clinical Manager Great Ormond Street Hospital for Children NHS Foundation Trust
  • Other Contributors:
    • Jo Wray, Health Psychologist
    • Kate Oulton, Research Fellow
    • Samiran Ray, Research Fellow
    • Mark J Peters, Professor of Paediatric Intensive Care

  We are pleased to see the interest shown in our article by Drs Cheung and Lachman, but cannot agree with their assertion that our research ‘misses the point’.
  Despite widespread use, there remains limited research on the effectiveness of paediatric early warning systems (PEWS) in detecting deterioration in hospitalised children. Our paper sought to establish if there were statistically significance differences in performance between 18 published systems. Trigger systems were out-performed by scoring-systems in this relevant but narrow assessment. Our conclusion emphasizes that it is unclear what factors account for these differences in performance.
  Dr Cheung and others feel this observation of statistical inferiority of trigger system is not merited and the observed differences may be influenced by the scoring threshold selected. Dr Cheung illustrates this by comparison to the threshold selection of the C-reactive protein test as an indicator of inflammation. We found this to be rather confusing. The outcomes of trigger-based systems are, by their very nature, dichotomous. Whilst there is always a trade-off between sensitivity and specificity for scoring-based systems, the same does not apply to trigger-based systems. The system is either triggered or not. We note that Dr Cheung does not offer data to support his preference for trigger systems.
  We agree that managing deterioration in children is complex. However it is hard to imagine how this would be im...

  Show More

  We are pleased to see the interest shown in our article by Drs Cheung and Lachman, but cannot agree with their assertion that our research ‘misses the point’.
  Despite widespread use, there remains limited research on the effectiveness of paediatric early warning systems (PEWS) in detecting deterioration in hospitalised children. Our paper sought to establish if there were statistically significance differences in performance between 18 published systems. Trigger systems were out-performed by scoring-systems in this relevant but narrow assessment. Our conclusion emphasizes that it is unclear what factors account for these differences in performance.
  Dr Cheung and others feel this observation of statistical inferiority of trigger system is not merited and the observed differences may be influenced by the scoring threshold selected. Dr Cheung illustrates this by comparison to the threshold selection of the C-reactive protein test as an indicator of inflammation. We found this to be rather confusing. The outcomes of trigger-based systems are, by their very nature, dichotomous. Whilst there is always a trade-off between sensitivity and specificity for scoring-based systems, the same does not apply to trigger-based systems. The system is either triggered or not. We note that Dr Cheung does not offer data to support his preference for trigger systems.
  We agree that managing deterioration in children is complex. However it is hard to imagine how this would be improved through the use of a tool with reduced ability to discriminate children ‘at risk’ of deterioration, even if staff used this optimally. Or how the use of a tool with an unknown performance would improve the situation. Dr Cheung states “One would not recommend a new drug be widely used, regardless of its efficacy, without also evaluating whether the means of administration was reliable and acceptable.” We respectfully suggest that understanding whether an intervention is efficacious is a fundamental first step. There is no sense in evaluating the reliability and acceptability of ineffective interventions. The real world clinical effectiveness does indeed depend on many factors including human behaviour. However, clinicians are unlikely to adopt a system enthusiastically that does a poor job of categorising patients as high or low risk of deterioration.
  Dr Cheung is correct in stating that score-based systems are inherently more complex to use, however this can be over-come by adopting ‘smart’ technology to automatically calculate scores, as we have done in our institution. As technology continues to develop it may be time for healthcare to work ‘smarter’ by embracing technological solutions to problems rather than demanding staff work ‘harder’. Neither quality nor its improvement should be constrained by the systems within which processes occur. While quality can be optimized within a system, improvement can be stunted.
  Our paper sought to establish if there were differences in performance between differing systems and whether these were significant. Our findings indicate that differences do indeed exist and as such, this may be an important consideration for centres using a PEWS. As the editorial by Lillitos and Maconochie emphasises – it is time to focus on the evidence.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 3 August 2017

  Content vs Context in scoring systems

  • Damian T Roland, Honorary Associate Professor in Paediatric Emergency Medicine On behalf of the PUMA Study group (https://pumastudy.com/)

  The need to improve outcomes in child health is not disputed, especially in the UK where performance lags behind the rest of Europe [1]. Mechanisms which detect, and respond, to the deteriorating child in an effective manner should be validated and shared so we welcome further research by Chapman et al. [2] which demonstrates the complexity of producing tools which do this. However, we support the concern raised by Cheung and Lachman [3] in ensuring that appropriate conclusions are drawn from this work. As a research group funded to investigate these systems, from both a quantitative and qualitative viewpoint, we would like to highlight some of the dangers in use of terminology in this area. The concept that systems may be a better paradigm than scores (i.e. the amalgamation of observations into binary or composite measures which determine pre-defined actions) is not new [4] and it is already recognised that both afferent and efferent limbs are vital in order to complete what Joffe described as the, “chain of events needed to improve response to inpatient deterioration.” We note Maconochie and Lillitos use the term PES (Paediatric Early Warning System) and differentiate trigger systems from PEWS (Paediatric early warning scores). It is not however clear why trigger systems are treated as separate from PEWS as the literature contains numerous examples of where a trigger type system has been labelled as a PEWS [5]. While we entirely agree there is a challenge in labelling...

  Show More

  The need to improve outcomes in child health is not disputed, especially in the UK where performance lags behind the rest of Europe [1]. Mechanisms which detect, and respond, to the deteriorating child in an effective manner should be validated and shared so we welcome further research by Chapman et al. [2] which demonstrates the complexity of producing tools which do this. However, we support the concern raised by Cheung and Lachman [3] in ensuring that appropriate conclusions are drawn from this work. As a research group funded to investigate these systems, from both a quantitative and qualitative viewpoint, we would like to highlight some of the dangers in use of terminology in this area. The concept that systems may be a better paradigm than scores (i.e. the amalgamation of observations into binary or composite measures which determine pre-defined actions) is not new [4] and it is already recognised that both afferent and efferent limbs are vital in order to complete what Joffe described as the, “chain of events needed to improve response to inpatient deterioration.” We note Maconochie and Lillitos use the term PES (Paediatric Early Warning System) and differentiate trigger systems from PEWS (Paediatric early warning scores). It is not however clear why trigger systems are treated as separate from PEWS as the literature contains numerous examples of where a trigger type system has been labelled as a PEWS [5]. While we entirely agree there is a challenge in labelling both scores and systems together we are not certain that the addition of a new term, PES, will aid this clarification. The PUMA study [6] is investigating the ‘multifaceted sociotechnical system’ suggested in the recent systematic review by Lambert [5] and we, in agreement with Cheung and Lachman feel that simply comparing scoring systems against each other risks missing the point that it may be the contextual ‘system’ as a whole, and not the particular form of a tool, that has the biggest impact on patient outcomes.

  1. Wolfe I, Donkin A, Marmot M, et al. UK child survival in a European context: recommendations for a national Countdown Collaboration. Archives of Disease in Childhood 2015;100:907-914.
  2. Chapman SM et al. ‘The Score Matters’: wide variations in predictive performance of 18 paediatric track and trigger systems. Arch Dis Child 2017; 102:487-95
  3. Cheung R and Lachman P. "Human factors matter" - Statistical analysis of performance of trigger systems misses the point. In response to: Lillitos PJ, Maconochie IK. Paediatric early warning systems (PEWS and Trigger systems) for the hospitalised child: time to focus on the evidence. Arch Dis Child 2017;102: 479-80
  4. Joffe AR, Anton NR, Burkholder SC. Reduction in hospital mortality over time in a hospital without a pediatric medical emergency team: limitations of before-and-after study designs. Arch Pediatr Adolesc Med 2011;165:419–23.
  5. Lambert V, Matthews A, MacDonell R, et al. Paediatric early warning systems for detecting and responding to clinical deterioration in children: a systematic review BMJ Open 2017;7:e014497
  6. PUMA - Paediatric early warning system (PEWS): Utilisation and Mortality Avoidance. A prospective, mixed methods, before and after study identifying the evidence base for the core components of an effective PEWS and the development of an implementation package for implementation and use in the UK HS&DR 12/178/17 https://www.journalslibrary.nihr.ac.uk/programmes/hta/1217817#/ Website last accessed 20th June 2017

  Show Less

  Conflict of Interest:
  Our research group is delivering an NIHR Funded HS&DR programme to produce an evidence-based, system-wide approach to improving paediatric patient safety and reducing mortality in UK hospitals.

  • Back to top
• Published on: 19 June 2017

  "Human factors matter" - Statistical analysis of performance of trigger systems misses the point

  • Ronny Cheung, Consultant Paediatrician Evelina London Children's Hospital, Guy's & St Thomas' Foundation NHS Trust
  • Other Contributors:
    • Peter Lachman, Consultant Paediatrician & CEO

  Chapman et al (1) present a valuable evaluation of the performance characteristics of 18 commonly used paediatric early warning systems. They observed that the performance of the 12 “scoring” systems (where cumulative component values for vital signs are used to identify thresholds for escalation of care) was superior to 6 “trigger” systems (where breaching set thresholds for one or more vital signs lead to escalation without the need for adding numerical scores), based on sensitivity, specificity and area under the operating curve (AUROC). Although they do not specifically claim that this finding should be extrapolated to suggest that all scoring systems outperform trigger systems, this is the implication both in the results and discussion section. Indeed, the associated editorial by Lillitos & Maconochie confirms this implied conclusion, when they state that “In conclusion…overall, PEWS perform better than Trigger systems.” (2)

  We contend that this is an erroneous and misleading conclusion and far outstrips the scope and methodology of the study. Firstly, the findings are related to the performance of 16 specific tools and no comment can be made about whether it is the Trigger or the Score aspects which are responsible for this difference. Using the analogy of a therapeutic trial, there can be no basis to conclude that this is a “class effect” rather than specific to each tool. Secondly, the authors themselves recognise that it is the thresholds for escalation...

  Show More

  Chapman et al (1) present a valuable evaluation of the performance characteristics of 18 commonly used paediatric early warning systems. They observed that the performance of the 12 “scoring” systems (where cumulative component values for vital signs are used to identify thresholds for escalation of care) was superior to 6 “trigger” systems (where breaching set thresholds for one or more vital signs lead to escalation without the need for adding numerical scores), based on sensitivity, specificity and area under the operating curve (AUROC). Although they do not specifically claim that this finding should be extrapolated to suggest that all scoring systems outperform trigger systems, this is the implication both in the results and discussion section. Indeed, the associated editorial by Lillitos & Maconochie confirms this implied conclusion, when they state that “In conclusion…overall, PEWS perform better than Trigger systems.” (2)

  We contend that this is an erroneous and misleading conclusion and far outstrips the scope and methodology of the study. Firstly, the findings are related to the performance of 16 specific tools and no comment can be made about whether it is the Trigger or the Score aspects which are responsible for this difference. Using the analogy of a therapeutic trial, there can be no basis to conclude that this is a “class effect” rather than specific to each tool. Secondly, the authors themselves recognise that it is the thresholds for escalation that are intrinsic to sensitivity and specificity measurements, and that changes in those thresholds might ameliorate the apparent difference in performance between all 18 tools. This is unrelated to whether each tool is a Trigger or Score systems. One would not judge C-reactive protein to be a uniformly poorly specific test of inflammation based on a threshold of <5 mg/L – rather, one would assess the threshold of significance.

  Thirdly, and most importantly, the misleading implication that Score systems outperform Trigger tools based purely on test performance misses the point – that early warning systems are a multicomponent intervention and not just restricted to the scoring system in isolation. Chapman et al’s study assesses only the statistical performance in but not overall effectiveness. One would not recommend a new drug be widely used, regardless of its efficacy, without also evaluating whether the means of administration was reliable and acceptable. The recognition of deteriorating patients depends on correct recording, identification of when a threshold is reached, and not just whether the threshold is correct. There is evidence that Score-based tools are subject to significantly greater errors in completion and interpretation than Trigger tools, due to the complexity of calculations and human error. (3,4) These error rates are likely to be even more exaggerated in live clinical use than in controlled experimental settings. Future widespread implementation of electronic systems which automate the calculation process may eliminate this issue, but this is not yet widely available. Since the research was undertaken in 2011-12, there have been advances in the understanding on what is required to detect deterioration and that trigger tools are only one part of the intervention. There has also been more research on trigger and track systems and their implementation. The effectiveness of any of the tools will be dependent on the context in which the tool is used. (5,6)

  Given the increasing recognition that human factors play a significant role in patient safety interventions and especially the recognition and escalation of deterioration, it is important to clarify the limitations of the findings from this study, to ensure they are not misinterpreted to imply that Score-based tools are inherently superior to Trigger systems, especially in “live” use. A recent review of PEWS concluded that “future research needs to investigate PEWS as a complex multifaceted sociotechnical system that is embedded in a wider safety culture influenced by many organisational and human factors.” (7) This should be the focus of research rather than concentrating on PEWS as a tool evaluated mainly by statistical performance. Neither the paper by Chapman et al (1) nor the accompanying editorial (2) appears to take this complexity into account.

  References:
  1. Chapman SM et al. ‘The Score Matters’: wide variations in predictive performance of 18 paediatric track and trigger systems. Arch Dis Child 2017; 102:487-95.
  2. Lillitos PJ, Maconochie IK. Paediatric early warning systems (PEWS and Trigger systems) for the hospitalised child: time to focus on the evidence. Arch Dis Child 2017;102: 479-80
  3. Christofidis MJ et al. A human factors approach to observation chart design can trump health professionals’ prior chart experience. Resuscitation 2013; 84: 657-665
  4. Preece MHW et al. Supporting the detection of
  patient deterioration: observation chart design affects the recognition of abnormal
  vital signs. Resuscitation 2012;83:1111–8.
  5. Brady et al,. Improving Situation Awareness to Reduce Unrecognized Clinical Deterioration and Serious Safety Events. Pediatrics, 2013. 131; e298-e308
  6. Hughes C, Pain C, Braithwaite J, et al. ‘Between the flags’: implementing a rapid response system at scale
  BMJ Qual Saf 2014;23:714-717.
  7. Lambert V, Matthews A, MacDonell R, et al. Paediatric early warning systems for detecting and responding to clinical deterioration in children: a systematic review BMJ Open 2017;7:e014497. doi: 10.1136/bmjopen-2016-014497

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top