Article Text
PDF
Abstract
Objective The authors aimed to assess the repeatability of the Manchester Triage System (MTS) in children.
Methods All emergency department nurses (n=43) from a general teaching hospital and a university children's hospital in The Netherlands triaged 20 written case scenarios using the Manchester Triage system. Second, at two emergency departments (EDs), real-life simultaneous triage of patients (<16 years) was performed by ED nurses and two research nurses. The written case scenarios and the patients included in the real-life simultaneous triage study were representative of children attending the ED, in age, problem and urgency level. The authors assessed inter-rater agreement using quadratic weighted kappa values.
Results The weighted kappa between the nurses, triaging the case scenarios, was 0.83 (95% CI 0.74 to 0.91). In total, 88% (N=198) of the eligible ED patients were triaged simultaneously, with a weighted κ of 0.65 (95% CI 0.56 to 0.72).
Conclusions The MTS showed good to very good repeatability in paediatric emergency care.
- Triage
- emergency service
- hospital
- children
- repeatability
- reliability
- emergency care systems
- nursing
- emergency departments
- paediatrics
Statistics from Altmetric.com
- Triage
- emergency service
- hospital
- children
- repeatability
- reliability
- emergency care systems
- nursing
- emergency departments
- paediatrics
Introduction
As triage aims to see patients first who benefit most from immediate care, it is essential that triage be both objective and reproducible. Different triage systems are extensively used in emergency departments (EDs) across the world. The Manchester Triage System (MTS) was described and published in 1997 and is nowadays adopted around the world.1 2 Little research on repeatability and validity of triage systems in paediatric emergency care has been conducted to date.3–9 As triage systems are widely used, and it is not yet clear if one system is preferred over the others, research on their repeatability and validity is important and must be performed.
The MTS was developed by expert opinion.1 The Dutch Institute of Healthcare recommended using the MTS in The Netherlands.10 It consists of 52 flow charts all representing a presenting problem, of which 49 are suitable for children. Following flow-chart selection, general (life threat, haemorrhage, pain, conscious level, temperature and acuteness) and specific discriminators are considered. For example, a patient with an affirmative response to the discriminator ‘Increased work of breathing?’ is triaged into urgency level 2. Patients are allocated into one of five urgency levels. The MTS prescribes the maximum waiting time for each urgency category (0, 10, 60, 120 and 240 min).
In adults, the MTS was shown to be sensitive for those with chest pain (sensitivity 87%, 95% CI 78 to 92 and specificity 72%, 95% CI 61 to 82 to identify high-risk cardiac chest pain)11 and for those with a critical illness.12 The Manchester pain scale, a part of the MTS, showed a strong concurrent validity when compared with the Oucher pain scale.13 The inter-rater agreement of the MTS in all ages demonstrated a quadratic weighted κ of 0.62 (95% CI 0.60 to 0.65) when studied using written case scenarios.14 In a large prospective observational study, the MTS demonstrated moderate validity when used in paediatric emergency care. It errs on the safe side, with much more overtriage than undertriage compared with an independent reference standard for urgency.8 9 The inter-rater agreement of the MTS for children in particular has not yet been evaluated.
The aim of this study was to evaluate repeatability of the MTS in paediatric emergency care, using both written case scenarios and simultaneous triages by ED nurses.
Methods
Study design
To study repeatability, we performed two studies on inter-rater agreement. First, 20 written case scenarios were triaged by 43 ED nurses, from two different hospitals, using the MTS (Part 1). Second, 198 patients presenting to the two study EDs were each triaged simultaneously using the MTS, by one out of 25 ED nurses and one out of two research nurses (Part 2). Table 1 reviews our study design. The requirement for informed consent was waived by the institutional review board.
Study design
Patients
The ED of the Erasmus University Medical Center-Sophia Children's Hospital, Rotterdam is a paediatric-specific ED and is visited by nearly 9000 patients per year. The MTS was implemented in 2005. The ED of the Haga Hospital-Juliana Children's Hospital, The Hague is a general paediatric–adult ED in a large teaching hospital with approximately 30 000 patients visits yearly, including 15 000 paediatric visits. For this site, the MTS was implemented in 2003. Participating ED nurses were experienced in both paediatric nursing and ED nursing, with a median of 10 years of ED nursing experience (IQR 7–14 years) and a minimum of 2 years. Both studies were performed between November 2006 and February 2007.
Manchester Triage System
Children under 16 years of age visiting the ED were triaged using a computerised version of the MTS. Registered nurses selected an MTS flow chart that suits the problem the patient presents with. Selection of the appropriate discriminator leads to allocation of an urgency level. The chosen flow chart and discriminator were documented by the software application during triage. We used the official, translated version of the MTS advocated by the Dutch Association of ED Nurses.1 15 Triage difficulties identified by the nurse participants could be reported and were discussed at ED meetings.
Part 1: written case scenarios
Twenty written case scenarios were obtained and translated from Baumann et al.3 Case scenarios are based on children presenting to the emergency department. Age, gender and presenting symptoms of the case scenarios were comparable with the total population presenting at the two EDs (table 2).
Patient characteristics of the total population presenting to the emergency departments in 2006 and the patients selected for the real life simultaneous triage (Part 2) and the written case scenarios (Part 1)
The high-urgency patients were over-represented; the cases contained more boys and were somewhat older.
Forty-four nurses received a written description of the cases and triaged the cases using the digital MTS application. Each case provided the patient's age, gender, problem of encounter and a short description of the history and vital signs. An example of a written case scenario (English translation) is as follows:An 8-year-old female presents to triage with her mom. The child has a sore throat, vomiting, and a fever all day. Mom states her child has been having difficulty swallowing all day. The child is making grunting noises and her skin is warm and flushed. T 38.7°C, HR 122/min, Resp Rate 22/min, BP 110/53, SpO2 99% on room air.
Part 2: real-time simultaneous triage
Patients attending the ED were triaged by one of 25 ED nurses. One of the two research nurses was present during the triage assessment but did not interfere. After the assessment, both nurses triaged the patient. Patients were included during 12 work shifts ranging in duration from 7 to 10 h. The research nurses selected the shift on basis of their own availability and were not aware of the working schedule of the triage nurses. They triaged all consecutive patients presenting at the ED.
Data on patient characteristics were gathered prospectively by the ED nurse in the triage application.
Primary data analysis
The characteristics of included patients were compared with characteristics of the total group of patients presenting at the same two EDs during 7 and 13 months respectively in 2006/2007.9 The agreement between the nurses in MTS urgency level, flow chart and discriminator was determined for all 20 cases. First, we considered the urgency, flow chart or discriminator with the highest percentage agreement between nurses per case, and second, we calculated the median and IQR of the percentage agreement of all cases. We determined the quadratic weighted κ (Kw) by calculating the intraclass correlation coefficient (ICC) for agreement in urgency level. The ICC is equivalent to the quadratic weighted κ.16 The quadratic weighted κ uses increasing weights for more severe disagreement.17 We used the two-way mixed model, type consistency function to calculate the ICC, for two as well as for multiple raters (SPSS 14.0.1 Chicago, Illinois, USA, SPSS, v. 8.2, College Station, Texas, USA). The simple κ was calculated for agreement in the chosen MTS flow chart and discriminator using Stata v 8.2.
κ Values can be interpreted as poor if κ<0.20, fair if >0.21<κ and <0.40, moderate if >0.41<κ and <0.60, good if >0.61<κ and <0.80, and very good if κ>0.80.17
Results
Part 1: inter-rater agreement: written case scenarios
All ED nurses (N=44) working at the two EDs triaged each scenario. The results from one nurse were excluded due to a procedural error of the computer application. As a result, data from 43 nurses were included, 24 from the university hospital and 19 from the general hospital. The median agreement in urgency level was 81% (IQR 60%, 90%) with a Kw of 0.83 (95% CI 0.74 to 0.91). For traumatic cases, the Kw was 0.91 (95% CI 0.80 to 0.98) and for non-traumatic cases 0.77 (95% CI 0.63 to 0.90).
Part 2: real-time simultaneous triage
During six shifts in December 2006 and six shifts in February 2007 (between 10 am and 6 pm or between 1 pm and 11 pm), 198 patients were triaged simultaneously (88% of eligible patients). One hundred and thirty-nine were included at the general hospital and 59 at the university hospital. No patients refused to participate. One research nurse was available per shift, and consequently some patients were missed because they entered the ED at the same time as other patients. The characteristics of the selected patients were comparable with the characteristics of the total ED population, except that the selected patient cohort contained slightly more patients with fever without a focus than the general patient population.
The agreement in MTS urgency level between the triage nurse and the research nurse was 66% with a Kw of 0.65 (95% CI 0.56 to 0.72). In most cases of disagreement in urgency level, the disagreement was one level (28%, N=56) (table 3).
Real-life simultaneous triage (Part 2): agreement in Manchester Triage System urgency level between the triage nurse and the research nurse
The used MTS flow chart and discriminator to triage patients and decide on urgency were available in 190 versus 181 patients, respectively.
The agreement in MTS flow chart and discriminator was 64% and 28% respectively, with simple κ scores of 0.60 (95% CI 0.55 to 0.64) and 0.26 (95% CI 0.23 to 0.29). Pain score was documented at triage in 60% of the cases, with nurses agreeing on pain score in 24% of cases, Kw 0.44 (95% CI 0.28 to 0.58). Disagreement in urgency level between triage nurse and research nurse was strongly related to disagreement in discriminator and not related to disagreement in flow chart. (agreement/disagreement in urgency vs agreement/disagreement in discriminator OR=(52/3)/(69/57),=14, 95% CI 4.2 to 48). (agreement/disagreement in urgency vs agreement/disagreement in flow chart, OR=(83/39)/(43/25)=1.2, 95% CI 0.7 to 2.3).
Nurses agreed in 66% in both patients with a traumatic problem (N=56, Kw 0.45, 95% CI 0.22 to 0.64) and patients with a non-traumatic presenting problem (N=142, Kw=0.60,95% CI 0.48 to 0.69). Disagreement did not depend on the patient's age (median age 2.47 and 2.69 years, Mann–Whitney test, p=0.55).
Discussion
This study showed adequate repeatability of the MTS when applied to paediatric emergency care. The MTS demonstrated good to very good inter-rater agreement when studied using written case scenarios and real-time simultaneous triage.
Compared with the inter-rater agreement of other triage systems studied in children using written case scenarios, the inter-rater agreement found for the MTS in our study is high (table 4).
Inter-rater agreement of triage systems for children
The agreement found in a simultaneous triage of the MTS is somewhat higher than with a simultaneous triage using the Emergency Severity Index (ESI)3 and the Pediatric Canadian Triage and Acuity Scale, in children,7 and lower than with the Soterion Rapid Triage System.19 However, the studies on the ESI and Pediatric Canadian Triage and Acuity Scale studies performed the triage assessment twice, which may explain a lower agreement.
In adults, the inter-rater agreement (weighted κ) of five-level triage systems studied by simultaneous triage ranged from 0.66 to 0.87.19–22 Two studies used written case scenarios and demonstrated a weighted κ of 0.80 and 0.71.22 23
In several studies weighted κ values were calculated to determine inter-rater agreement. However, from these papers, it is often not clear if linear or quadratic weighted κ values were calculated (table 4). A quadratic weighted κ gives a somewhat higher weight if raters disagree with only one level compared with the linear weighted κ. In our study, we determined quadratic weighted κ values.16
We argue that the inter-rater agreement of triage systems depends roughly on three criteria. First, nurses must be experienced with the signs and symptoms of patients presenting at the ED. Second, the nurses must be well trained in the particular triage system in order to use the correct definitions belonging to the discriminators. The nurses working in the studied hospitals all met these criteria.
Third, the triage system must be unambiguous and should contain discriminators numerous enough to match the diversity of patients visiting the ED. For example, one written case scenario had a very low agreement in urgency level, since nurses had chosen 14 different discriminators to triage the case. It presented a 2-month-old boy with a short period of apnoea. The fact that this presentation (incident or Apparent Life-Threatening Event, ALTE) is not exactly covered in the MTS probably explains the low agreement for this case.
Agreement could potentially be improved with ongoing training for ED nurses. After finishing the study, investigators discussed the disconcordant cases with the ED nurses in order to improve the triage process.
Our results showed that the agreement on flow-chart level (representing the patient's presenting problem) is moderate (κ 0.60, 95% CI 0.55 to 0.64) and on discriminator level fair (κ 0.26, 95% CI 0.23 to 0.29). The low agreement at discriminator level did not result in a low agreement in urgency level. Since more MTS discriminators can lead to one urgency level, the low agreement in MTS discriminator has little influence on the urgency level. This provides evidence supporting a high internal consistency for the MTS.
In the case scenarios (part 1), nurses performed a higher agreement in traumatic cases compared with non-traumatic cases. However, 0.91 and 0.77 both represent a very good and good inter-rater agreement. The difference between good and very good agreement is not considered clinically important.
To appreciate the results, some limitations should be considered.
The set of written case scenarios was obtained from another study group, so we had no influence on the selection of cases. That is why selection bias does not seem likely. To check if the cases were representative of our population, we compared patient characteristics of the cases with our population (table 2). The cases were comparable with our population.
The triage of written case scenarios is not an exact substitute for evaluation of the actual triage process. The nuance of the nurse's interpretation of each patient's signs and symptoms is an important part of the triage process, and this essence is not captured using the written case scenarios method. We attempted to address this shortcoming of the paper scenarios with the addition of real-time simultaneous triages. This method still demonstrated a good inter-rater agreement.
The written case scenario method is often used to assess the inter-rater agreement of triage systems. A recent study showed moderate to high agreement between simultaneous triage and paper case scenarios.24
During the real-time simultaneous triage (part 2), we did not perform the triage assessment twice. The research nurse was present during the assessment of the triage nurse. Subsequently, both nurses triaged the patient blinded using the MTS in a separate room and did not discuss the patients' signs and symptoms with each other.
A double independent triage assessment might better evaluate the actual triage process. However, such a method was not possible because of the possible impact on patient management and waiting times. Using a double assessment method, the nurse's translation from the patient's signs and symptoms to a triage decision would be incorporated.
Conclusion
The MTS has a good to very good inter-rater agreement when applied to paediatric emergency patients. Good repeatability is an essential requirement for valid triage.
What is already known about this topic
The Manchester Triage System is often used at emergency departments in Europe.
The validity of the system was evaluated in adult patients with chest pain and patients with critical illness.
Although other triage systems showed a moderate to good repeatability, the repeatability of the MTS has not been evaluated for children, in particular.
What this study adds
We reported a good to very good inter-rater agreement (repeatability) for the Manchester Triage System when applied to children at the emergency department.
Acknowledgments
We thank the ED nurses of the Erasmus MC—Sophia Children's Hospital and the Haga Hospital-Juliana Children's Hospital for their cooperation in this study, M de Wilde, B. Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands for technical support, and MJC Eijkemans, Department of Public Health, Erasmus MC, University Medical Center, Rotterdam, The Netherlands for statistical support.
References
Footnotes
Funding This study was financially supported by Zon-MW, The Netherlands Organization for Health Research and Development, and Erasmus MC, University Medical Center Rotterdam, The Netherlands.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.