Aims: To evaluate the performance of the Paediatric Risk of Mortality (PRISM) score in a population of UK children and to use this score to examine severity of illness adjusted mortality of critically ill children <16 years old in a defined geographical region.
Methods: Observational study of a defined population of critically ill children (<16 years old) admitted to hospitals in the South West Region between 1 December 1996 and 30 November 1998.
Results: Data were collected from 1148 eligible admissions. PRISM was found to perform acceptably in this population. There was no significant difference between the overall number of observed deaths and those predicted by PRISM. Admissions with mortality risk 30% or greater had significantly greater odds ratio for death in general intensive care units compared with the tertiary paediatric intensive care unit.
Conclusions: Children with a high initial risk of mortality based on PRISM score were significantly more likely to survive in a tertiary paediatric intensive care unit than in general intensive care units in this region. However, there was no evidence from this study that admissions with lower mortality risk than 30% had significantly worse mortality in non-tertiary general units than in tertiary paediatric intensive care units.
- intensive care
- hospital mortality
- outcome assessment
- BPA, British Paediatric Association
- ICU, intensive care unit
- PICU, paediatric intensive care unit
- PIM score, Paediatric Index of Mortality score
- PRISM score, Paediatric Risk of Mortality score
- ROC, receiver operator characteristic
- TISS, therapeutic invervention scoring system
Statistics from Altmetric.com
- BPA, British Paediatric Association
- ICU, intensive care unit
- PICU, paediatric intensive care unit
- PIM score, Paediatric Index of Mortality score
- PRISM score, Paediatric Risk of Mortality score
- ROC, receiver operator characteristic
- TISS, therapeutic invervention scoring system
There has been a long running debate about the optimal configuration of paediatric intensive care services in the UK. In 1993, a report by the British Paediatric Association (BPA) highlighted the fragmentation of paediatric intensive care facilities in the UK. The report indicated that a substantial proportion (20.5%) of critically ill children were treated in adult, general intensive care units (ICUs) which admitted, on average, only 21 children per year, rather than dedicated paediatric intensive care units (PICUs).1 The BPA survey did not show differences in mortality between critically ill children treated in different types of units (ICUs versus PICUs), but the lack of a difference in observed mortality was attributed to probable differences in case mix (age, acute severity, co-morbidity, reason for admission) between the two populations.
In 1988, Pollack et al developed a risk adjustment method, the Paediatric Risk of Mortality (PRISM) score, based on acute physiological derangement, to facilitate comparisons of mortality between different populations of critically ill children.2 In 1991, using this method of risk adjustment, Pollack et al showed that the odds of dying were significantly higher (OR 8.11) for the most severely ill children (an initial PRISM probability exceeding 30%) treated in non-tertiary ICUs compared with those treated in tertiary PICUs.3 The PRISM score was subsequently validated in an independent European population in the Netherlands,4 and a similar comparison of outcomes from tertiary and non-tertiary units again showed a significant odds ratio of death of 2.45 for the most severely ill children (initial PRISM probability exceeding 15%) treated in non-tertiary PICUs compared with those treated in a tertiary PICU.5 PRISM has been evaluated in single unit studies in the UK,6, 7 but no multicentre validation has been published and no equivalent studies of risk adjusted outcomes between different categories of ICUs have been undertaken. However, evidence has been found for higher adjusted mortality of children treated in one region of the UK, an example of a fragmented children's intensive care service, compared with Victoria in Australia where children's intensive care is centralised in one hospital.8 Using a different risk adjustment instrument, the Paediatric Index of Mortality (PIM) score,9 Pearson et al showed an adjusted odds ratio for death in Trent of 2.09 compared with Victoria. Gathering evidence for improved outcomes for critically ill children cared for in dedicated PICUs compared with non-tertiary units and a consensus of expert opinion led to the publication in 1997 of a strategic framework document for the reorganisation of British paediatric intensive care into a small number of large units providing intensive care services to children on a regional basis.10 However, the debate about optimal configuration of services for critically ill children has continued.11, 12
The present study (South West Critically Ill Children Study) was established in 1996 as a two year prospective study that aimed to identify all critically ill children in a defined geographical region regardless of their place of care. The objectives of the study were to evaluate the performance of the PRISM score in a multicentre population of UK children and to examine the relation between risk adjusted mortality and place of care in this population. The principal hypotheses of this study were that the PRISM score would fit the data acceptably for risk adjustment of this population and that there would be no difference in risk adjusted mortality using PRISM score between children treated in tertiary PICU and those treated in non-tertiary ICUs. A supplementary hypothesis was that there would be no difference in mortality between patients in different categories of mortality risk, based on Pollack and colleagues' risk stratification categories,3 treated in tertiary and non-tertiary units.
The geographical region that was chosen for our study was based on the administrative South West Region of the National Health Service, as it existed at the time the study commenced. This region encompassed 10 district general hospitals that admitted children, including two with specialist neurosurgical and burns units, and one tertiary children's hospital with a PICU. Ethics approval was obtained from the local research ethics committee of each hospital taking part.
All critically ill children, less than 16 years old, admitted to hospitals in the South West Region between 1 December 1996 and 30 November 1998 were identified. The definition of critical illness was based on a number of clinical and intervention criteria that were developed for the study (table 1). The criteria were designed to identify children with acute critical illness, whether or not they were admitted to an intensive or high dependency care area. A pilot study of these inclusion criteria was performed in an independent population at Royal Berkshire Hospital by one of us (SE). At this stage, the conditional criteria were modified to ensure that all children were identified who required intensive therapy or monitoring, but large numbers of children with very low risk of mortality on the basis of their PRISM score were excluded. A decision was made, a priori, to exclude: children who were admitted to hospital for elective cardiac surgery, as this took place at only one hospital in the region; infants who had never been discharged from hospital after birth, most of whom were treated in neonatal ICUs; and children who were not admitted to an intensive or high dependency care unit and had a low (less than 1%) PRISM probability of death. Children were also excluded whose defining illness was one of multiple admissions for the same chronic problem, such as chronic renal failure or treatment for oncological or haematological disease.
The lead research nurse (LG) visited every department that admitted children in each hospital in the region before the study commenced. In each hospital, a principal nurse was identified who acted as the liaison staff member for the study. The entry criteria were explained and the study documentation was reviewed. Attempts were made to ensure that the criteria were uniformly applied and examples were given to clarify individual criteria. For instance, “fits lasting for >1 hour” were seizures that had been continuing for one hour or had not stopped completely after treatment within one hour of their initiation, and “ventilator support” included any assisted respiration, such as bag and mask ventilation, but not elective anaesthesia procedures. For conditional criteria, apnoeas were defined as cessation of breathing for more than 20 seconds in association with observed physiological disturbance such as tachycardia or reduced oxygen saturation; continuous intravenous infusion of drugs included any drug other than an analgesic used for pain relief alone.
To check the completeness of recruitment, the paediatric admissions records of each hospital or unit were reviewed at regular intervals. Patients who would have potentially fulfilled our entry criteria were identified from the discharge diagnosis and a review was carried out on a 20% random sample from each hospital of these subjects. The entry criteria were applied retrospectively to information available from the notes to identify children who were eligible for inclusion in our study but who were not notified to the study coordinator.
Data collection and management
When a child met the entry criteria for the study, the local nurses informed the study coordinator in Bristol. This allowed the coordinator to track individual children and to ensure that data were returned for each child enrolled in the study. Study documentation was distributed to each unit in the region. This included a demographic data sheet that contained details of age, date of admission, diagnosis, and source of each patient. Each admission was given a unique study number and names were not recorded or stored. Admission and final diagnoses were categorised according to a hierarchical system developed by the Intensive Care National Audit and Research Centre.13 A separate sheet was used to record the actual values of physiological and biochemical data required to compute the PRISM score for each child. Where appropriate, the highest and lowest values during the first 24 hours were recorded. These measurements were recorded with reference to the first 24 hours after critical illness was recognised and for the first 24 hours after a child was moved to a new location of care; for example, ward to intensive care or between hospitals or units. For this study we have used the PRISM data from the first 24 hours as the risk adjustment method. Clinical staff were also asked to complete a daily record of therapeutic interventions using a therapeutic intervention scoring system (TISS).14 Study forms were returned to the coordinating centre when the index child was discharged from hospital or when the child died. The principal outcome measure we collected was death before hospital discharge and, for intensive care admissions, death before discharge from intensive care.
The mortality risk for each child was calculated from the PRISM score using the logistic regression model developed by Pollack and colleagues2 in a North American population:Solving equation (1) for r, . The goodness of fit of PRISM was tested by fitting receiver operator characteristic (ROC) curves and by Hosmer and Lemeshow's goodness of fit test.15 These test discrimination and calibration of the model respectively. As PRISM was originally validated for ICU deaths only, we tested the model for ICU admissions and deaths as well as for the whole population, which included non-ICU admissions.
To compare risk adjusted mortality between children in different categories of care (PICU versus ICU) we used logistic regression models. Two regression models were developed: model 1 included all children in PICU/ICU and model 2 included only those with a PRISM probability of death exceeding 1%. The models were adjusted for maximum TISS and organ system of primary diagnosis.
As Pollack et al had shown an apparent interaction between severity of illness and place of care on mortality in their study in the United States,3 the same risk categories based on PRISM mortality risk (<5%, 5–30%, >30%) were applied to the population in the present study and the effect of place of care on risk adjusted mortality was analysed separately for each category. The number of subjects required for this study was based on a prior power calculation to detect a significant difference in the probability of death between tertiary and non-tertiary ICU of 25% in the highest risk category (>30%), assuming the mortality in tertiary units in this category would average 60%. On this basis, 1500 subjects were required, assuming risk stratification proportions equivalent to those observed in our pilot data.
All analyses were performed using SPSS for Windows, version 9.
Data were collected on 2149 admissions during the two year study period. Figure 1 shows exclusions, resulting in a study population of 1148 children meeting the inclusion criteria. The 20% random sample identified 13 children who were eligible but not included in the study, suggesting that about 3% of eligible children were missed (13×5/2149). Figure 1 also shows the place of treatment of the population. A total of 438 children were treated in the PICU (168 admitted directly and 270 transferred from another hospital after critical illness identified), 375 children were treated in a general ICU, and 335 children were not admitted to ICU during their admission. No children were transferred from PICU to general ICU.
The median age of the 1148 children fulfilling the entry criteria was 22 months (range 3 days to 15 years 11 months); there were 661 boys (57.6%). The primary diagnosis was surgical in 206 admissions (18%) and non-surgical in the remainder. The principal organ systems involved were respiratory (36.9%), neurological (28.4%), haematological/immunological (including septicaemia) (10.2%), gastrointestinal (5.3%), and cardiovascular (5.1%), which together comprised 85.9% of admissions. The overall mortality rate was 7.4% (85 deaths) compared with 8.2% (94.1 deaths) (95% CI 83.5 to 104.7) predicted by PRISM.
Goodness of fit of PRISM
The area under the ROC curve of PRISM for ICU mortality (95% CI) was 0.82 (0.76 to 0.88). The same analysis for hospital deaths, as opposed to ICU death, for all admissions (ICU and non-ICU) had an identical area under the ROC curve of 0.82 (0.76 and 0.88). Table 2 shows the Hosmer and Lemeshow goodness of fit test for ICU admissions and ICU deaths. This showed no significant heterogeneity between observed and predicted deaths across 10 deciles of risk based on PRISM score (χ2 = 8.27 (8 df), p = 0.41). The same test applied to all admissions also gave a non-significant result (χ2 = 12.63 (8 df), p = 0.12). Therefore, on the basis of these findings, the performance of PRISM was judged to be satisfactory as a risk adjustment method in this population.
Comparison of mortality by place of care
Table 3 shows the characteristics of the three populations of children admitted to PICU, general ICU (including neurosurgery and burns), and non-ICU admissions. There were significant differences in the median age, day 1 PRISM and TISS scores, and in the proportion of boys between groups of children completing their intensive care episodes in the different categories of care facilities. There were fewer deaths than predicted by PRISM in the PICU population but the difference was not significant. This appears to have been a result of significantly fewer deaths than predicted in the transferred population. There were also significantly fewer deaths than predicted in the non-ICU population.
Tables 4 and 5 show the odds ratios from the logistic regression models for ICU admissions only. The number of deaths in the non-ICU population was smaller and the odds ratios were not significant. Table 4 shows the results of fitting models 1 and 2 to the ICU population, categorised as PICU or general ICU admissions. These showed significant interactions between place of care (PICU versus general ICU) and PRISM score. Figure 2 shows the interaction; the probability of death in general ICUs was comparable with or lower than the probability of death in PICU at low PRISM scores. However, admissions with high PRISM scores had a higher probability of dying in general ICUs compared with PICU. To investigate this further, the data were split into three mortality risk groups (<5%, 5–30%, >30%) and the odds ratios for dying according to place of care were analysed separately for each category (table 5). This showed that the overall odds of dying were not significantly different between PICU and general ICU. The odds of death were decreased for higher mortality risk categories within the PICU population, but these differences were not statistically significant. However, there was a significantly greater odds ratio of death in general ICUs compared with PICU (3.08, 95% CI 1.15 to 8.3) for children in the highest mortality risk stratum (>30%). This analysis was repeated using children in the middle risk category (5–30%) as the reference population and the odds ratio remained significant (2.91, 95% CI 1.06 to 8.0).
We have shown that critically ill children in the highest mortality risk stratum had a significantly greater risk of dying if they were treated outside the designated PICU serving a geographically defined population. These observations are consistent with those of studies in other countries that have examined mortality of children treated in ICUs.3, 5 Our study design differed from that of previous studies in our application of a population based approach to identifying critically ill children. This introduced some potential limitations to interpreting these findings. Our inclusion criteria were not comprehensive but represented an attempt to identify children with acute critical illness from a general population base. Therefore, significant omissions included children with oncological, renal, and cardiac diseases. The need for intensive care by some children in these populations is relatively predictable and their treatment tends to be concentrated on single tertiary centres. Therefore, their inclusion was likely to have biased the comparisons between secondary and tertiary care populations. There has also been concern that risk prediction scores, such as PRISM, may not be applicable to certain disease groups, such as children having cardiac surgery.7 Furthermore, the possibility of residual confounding of the results by differences in case mix between units cannot be excluded completely. An attempt was made in this study to control for case mix by including organ system of primary illness and PRISM, which takes account of age, operative status, and physiological stability in the mortality prediction equation, in the final model. However, we did not collect detailed information on co-morbidity and pre-existing medical conditions which thus remain a source of potential bias. Although it seems likely that children with organ specific, pre-existing morbidities would be more likely to be transferred at an early stage of their illness to a tertiary facility than previously healthy children, thus potentially increasing the observed to predicted mortality ratio in the tertiary PICU, this was not specifically addressed in the present study.
By relying on local reporting of cases, we are aware of the potential for omission of eligible admissions, particularly during busy clinical periods with rapid throughput of patients. This was addressed by conducting a random review of case notes, but there were anecdotal reports of potentially eligible admissions who were not notified to our study. Also excluded were children with low mortality risk who did not require high dependency or intensive care. As the principle outcome of this study was death or survival, the decision was taken to weight selection to the higher risk strata. Therefore, the denominator of the crude and adjusted mortality statistics is lower than the population at risk. However, overall mortality was broadly comparable to that reported by other groups.1, 8
The use of intervention measures in the selection criteria raises the possibility that some units may not have recognised some cases, and therefore failed to intervene appropriately, leading to non-notification of these cases. There was no evidence in these data of systematic bias between hospitals in the reporting of cases or in their risk stratification. However, the uniform application of criteria was not formally assessed in this study and this aspect needs to be considered if similar designs are used in future.
PRISM was used in this study as the risk stratification tool. This was largely for pragmatic reasons as PRISM was the most well validated score available when the study was designed. Since this study commenced, other risk prediction models have been developed (PIM,9 PRISMIII16) that have included measures of co-morbidity and, in the case of PIM, included UK children in the development9 and validation17 populations. PRISM was developed and validated in a North American population and its performance may not be applicable to the UK. However, to date there have been no multicentre comparisons of PRISM in a representative population of critically ill children, although single centre studies have criticised its applicability to some categories of intensive care admissions.6, 7 Also PRISM was validated as a predictor of intensive care death and the score is based on physiological and biochemical measurements within 24 hours of intensive care admission. This study included a proportion of critically ill children whose care episode did not include intensive care admission. Therefore, hospital death before discharge was used as the outcome and the PRISM score was calculated on variables measured during the first 24 hours after critical illness was recognised. The results of our evaluation of PRISM in this multicentre population showed acceptable ROCs when it was applied to intensive care admissions and deaths only, or to the whole study population. However, the calibration of PRISM was less good for all admissions compared with the intensive care population. The application of PRISM scoring prior to intensive care admission has been evaluated in one previous observational study of patients in four institutions in the United States; in three of these units, observed mortality corresponded well with predicted mortality based on preadmission PRISM.18
Another aspect of PRISM, which has also been observed by others, was under prediction of mortality. PRISM was first described in 1988 and it is likely that changes in the organisation and delivery of intensive care for children have led to improvements in outcome since then.19 This would be important in interpreting the results of this study if there was a systematic difference in the performance of PRISM between secondary care and tertiary care populations. No evidence for this was found in the present study, but this remains a potential source of confounding.
The principal finding of this study was a difference in mortality between admissions who completed their care episode in a tertiary PICU compared with those who were cared for in general ICUs. However, this applied only to those admissions with high initial risk of mortality (>30%). These results are consistent with the findings of Pollack et al in North America3 and Gemke et al in the Netherlands.5 Both groups showed significantly greater odds ratios for dying in non-tertiary units compared with tertiary PICUs. Furthermore, a study by Pearson et al, comparing Victoria, Australia with the Trent Region of the UK, showed a significantly greater odds ratio of dying in Trent, where treatment was delivered by a number of small units, than in Victoria, where care was centred on one large PICU.8 A recent study of North American units has also supported the provision of care in high volume, large units.20 This multicentre study showed a significant relation between patient volumes and risk adjusted mortality, although most (81%) of the units included were university affiliated, tertiary institutions and the extrapolation of these results to general ICUs in the UK remains to be confirmed.
Our study produced no evidence that children with a lower PRISM mortality risk than 30% had lower mortality in a tertiary PICU compared with a number of general ICUs in different hospitals. It is possible that children in these categories became more unstable after initial assessment and were selectively transferred to the tertiary unit where they subsequently died. However, mortality in the transferred population who completed care in the tertiary unit was lower than for other categories of admissions, including direct tertiary admissions, which tends to refute this explanation.
Fewer than predicted deaths were also observed in the admissions who were not admitted to intensive care during their hospital episode. The potential for selection bias is also present for this population. It is likely that children whose condition deteriorated after initial assessment on a ward would be moved to an intensive care area. There were few deaths of children in this category and it is possible that some of these deaths occurred in children whose underlying disease was deemed inappropriate for intensive care. Our study did not gather precise information about the decision making process or mode of dying, so this aspect has not been addressed. The pattern of mortality suggests that appropriate referral of most children to intensive care occurred. However, as noted by others in the UK,21 a large proportion of critically ill children in this study was not admitted to intensive care; it is of interest to note the wide range of mortality risk, and by inference physiological dependency, that admissions of these children presented to non-intensive care areas in general hospitals in our region during this study period.
The results of this study support the continued provision of care for the most physiologically unstable children in designated tertiary PICUs where they had a significantly decreased risk of death in our region. The place for tertiary care of children with less physiological derangement remains unclear on the basis of these results. Even within a population of critically ill children, death is a relatively uncommon outcome, and studies of morbidity and other markers of quality of care will be needed to address more subtle differences in outcomes between different units.
The study was the result of collaboration between several hospitals in the South West region and the authors wish to acknowledge their gratitude to all staff members who contributed to this study. Dr Andrea Sherriff, University of Bristol gave invaluable advice and assistance in the analysis of these data. The study was funded by a grant from the South West NHS Executive Research & Development Directorate. The following staff made particular contributions to the study: J Beer, Sister, ICU, Derriford Hospital, Plymouth; W Booth, Charge Nurse, PICU, Bristol Children's Hospital; Dr A Bosley, Consultant Paediatrician, North Devon District Hospital; Dr R Daum, Consultant Anaesthetist, ICU, Yeovil District Hospital; Dr A Day, Consultant Paediatrician, Cheltenham General Hospital; Dr A Daykin, Clinical Director ICU, Taunton & Somerset Hospital; Dr S Ferguson, Consultant Intensivist, Derriford Hospital, Plymouth; Dr N Gilbertson, Consultant Paediatrician, Royal Cornwall Hospital; Dr C Green, Consultant Intensivist, Gloucestershire Royal Hospital; L Hemington, Sister, ICU, Royal Devon & Exeter Hospital; V Holland, Sister, Paediatric Unit, Yeovil District Hospital; Dr L Jadresic, Consultant Paediatrician, Gloucestershire Royal Hospital; Dr I Jenkins, Director, PICU, Bristol Children's Hospital; N Lawrence, Nurse Manager, Torbay Hospital; Dr P MacNaughton, Clinical Director ICU, Derriford Hospital; A Maddox, Research Nurse, ICU, Royal Cornwall Hospital; Dr P Oades, Paediatrician, Royal Devon & Exeter Hospital; Dr J Purday, Consultant Anaesthetist, Royal Devon & Exeter Hospital, Plymouth; Dr R Sinclair, Consultant Intensivist, Royal Cornwall Hospital; S Syers, Sister, ICU, Derriford Hospital, Plymouth; Dr H Thomas, Paediatrician, Southmead Hospital, Bristol.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.