More information about text formats
Chapman et al (1) present a valuable evaluation of the performance characteristics of 18 commonly used paediatric early warning systems. They observed that the performance of the 12 “scoring” systems (where cumulative component values for vital signs are used to identify thresholds for escalation of care) was superior to 6 “trigger” systems (where breaching set thresholds for one or more vital signs lead to escalation without the need for adding numerical scores), based on sensitivity, specificity and area under the operating curve (AUROC). Although they do not specifically claim that this finding should be extrapolated to suggest that all scoring systems outperform trigger systems, this is the implication both in the results and discussion section. Indeed, the associated editorial by Lillitos & Maconochie confirms this implied conclusion, when they state that “In conclusion…overall, PEWS perform better than Trigger systems.” (2)
We contend that this is an erroneous and misleading conclusion and far outstrips the scope and methodology of the study. Firstly, the findings are related to the performance of 16 specific tools and no comment can be made about whether it is the Trigger or the Score aspects which are responsible for this difference. Using the analogy of a therapeutic trial, there can be no basis to conclude that this is a “class effect” rather than specific to each tool. Secondly, the authors themselves recognise that it is the thresholds for escalation...
We contend that this is an erroneous and misleading conclusion and far outstrips the scope and methodology of the study. Firstly, the findings are related to the performance of 16 specific tools and no comment can be made about whether it is the Trigger or the Score aspects which are responsible for this difference. Using the analogy of a therapeutic trial, there can be no basis to conclude that this is a “class effect” rather than specific to each tool. Secondly, the authors themselves recognise that it is the thresholds for escalation that are intrinsic to sensitivity and specificity measurements, and that changes in those thresholds might ameliorate the apparent difference in performance between all 18 tools. This is unrelated to whether each tool is a Trigger or Score systems. One would not judge C-reactive protein to be a uniformly poorly specific test of inflammation based on a threshold of <5 mg/L – rather, one would assess the threshold of significance.
Thirdly, and most importantly, the misleading implication that Score systems outperform Trigger tools based purely on test performance misses the point – that early warning systems are a multicomponent intervention and not just restricted to the scoring system in isolation. Chapman et al’s study assesses only the statistical performance in but not overall effectiveness. One would not recommend a new drug be widely used, regardless of its efficacy, without also evaluating whether the means of administration was reliable and acceptable. The recognition of deteriorating patients depends on correct recording, identification of when a threshold is reached, and not just whether the threshold is correct. There is evidence that Score-based tools are subject to significantly greater errors in completion and interpretation than Trigger tools, due to the complexity of calculations and human error. (3,4) These error rates are likely to be even more exaggerated in live clinical use than in controlled experimental settings. Future widespread implementation of electronic systems which automate the calculation process may eliminate this issue, but this is not yet widely available. Since the research was undertaken in 2011-12, there have been advances in the understanding on what is required to detect deterioration and that trigger tools are only one part of the intervention. There has also been more research on trigger and track systems and their implementation. The effectiveness of any of the tools will be dependent on the context in which the tool is used. (5,6)
Given the increasing recognition that human factors play a significant role in patient safety interventions and especially the recognition and escalation of deterioration, it is important to clarify the limitations of the findings from this study, to ensure they are not misinterpreted to imply that Score-based tools are inherently superior to Trigger systems, especially in “live” use. A recent review of PEWS concluded that “future research needs to investigate PEWS as a complex multifaceted sociotechnical system that is embedded in a wider safety culture influenced by many organisational and human factors.” (7) This should be the focus of research rather than concentrating on PEWS as a tool evaluated mainly by statistical performance. Neither the paper by Chapman et al (1) nor the accompanying editorial (2) appears to take this complexity into account.
1. Chapman SM et al. ‘The Score Matters’: wide variations in predictive performance of 18 paediatric track and trigger systems. Arch Dis Child 2017; 102:487-95.
2. Lillitos PJ, Maconochie IK. Paediatric early warning systems (PEWS and Trigger systems) for the hospitalised child: time to focus on the evidence. Arch Dis Child 2017;102: 479-80
3. Christofidis MJ et al. A human factors approach to observation chart design can trump health professionals’ prior chart experience. Resuscitation 2013; 84: 657-665
4. Preece MHW et al. Supporting the detection of
patient deterioration: observation chart design affects the recognition of abnormal
vital signs. Resuscitation 2012;83:1111–8.
5. Brady et al,. Improving Situation Awareness to Reduce Unrecognized Clinical Deterioration and Serious Safety Events. Pediatrics, 2013. 131; e298-e308
6. Hughes C, Pain C, Braithwaite J, et al. ‘Between the flags’: implementing a rapid response system at scale
BMJ Qual Saf 2014;23:714-717.
7. Lambert V, Matthews A, MacDonell R, et al. Paediatric early warning systems for detecting and responding to clinical deterioration in children: a systematic review BMJ Open 2017;7:e014497. doi: 10.1136/bmjopen-2016-014497