Objective: To evaluate the performance of the Paediatric Index of Mortality (PIM) in children cared for in adult intensive care units (ICUs) in district general hospitals in the South West Region of England.
Design and setting: An observational survey of all children admitted to adult ICUs in 15 district general hospitals between November 2000 and August 2002. For comparison, data were also collected from the regional paediatric ICUs between November 2000 and March 2002.
Results: Data were collected from 374 children admitted to adult ICUs and 850 children admitted to the regional paediatric ICU. There were significant differences in the patient characteristics between the two groups. In the adult ICU paediatric population, PIM discriminated well between death and survival (Az ROC = 0.96 (95% confidence interval, 0.93 to 0.99)) and calibrated well across deciles of risk (goodness of fit χ2 = 4.55 (8 df), p = 0.8).
Conclusions: PIM performs well as a risk adjustment method in children whose entire care remains in the adult ICU of a district general hospital. This is important should the Paediatric Intensive Care Audit Network (PICAnet) decide to extend its data collection beyond paediatric intensive care units to other units caring for critically ill children.
- PIM, Paediatric Index of Mortality
- ROC, receiver operator characteristic
- intensive care
- Paediatric Index of Mortality
Statistics from Altmetric.com
The Paediatric Index of Mortality (PIM) is a risk adjustment tool that uses physiological variables collected at the time of first contact of the patient with the paediatric intensive unit (PICU) to predict the probability of death. The use of PIM, which adjusts for differences in severity of illness and diagnosis, allows standards of care to be compared between units, and within units over time. Mortality prediction models such as PIM have been used to stratify patients appropriately in research trials and to compare different systems of health care delivery.1 The calibration of PIM has been assessed in United Kingdom paediatric intensive care units (PICUs),2 and has recently been updated to take into account changes in treatment and referral practice.3
The Paediatric Intensive Care Audit Network (PICAnet), supported by the Department of Health and based at the Trent Institute for Health Services Research, aims to establish a clinical database of paediatric intensive care activity that identifies best practice and facilitates strategic planning. PICAnet has chosen PIM as its preferred risk adjustment tool, and henceforth PIM/PIM2 variables are collected as part of the minimum core dataset from all PICUs. Recently, it has been proposed to extend this data collection exercise to adult ICUs that care for critically ill children. However, an important limitation of mortality prediction models is that they are only applicable to situations where the characteristics and diagnoses of the patients are similar to the original dataset from which the model was derived. Henceforth, in the case of PIM/PIM2 this translates to children managed in recognised paediatric intensive care units. It follows therefore that if PICAnet is usefully to be extended to include children cared for in adult ICUs, it will be necessary to make a proper evaluation of its performance in this distinct paediatric population. The aim of our study was to evaluate the performance of PIM in children cared for in adult ICUs in district general hospitals in the South West Region of England.
In the South West Region of the United Kingdom, data are routinely collected on all children who are admitted to adult intensive care units (ICUs) as part of the south west critically ill children’s audit. This data collection exercise allows the outcome of patients to be monitored, facilitates planning of paediatric services, and provided a dataset for this study. We conducted an observational survey of all children admitted to adult ICUs in 15 district general hospitals between November 2000 and August 2002. This study group encompassed children who remained on the ICU, and did not include those patients transferred to the lead PICU. For comparison, data were also collected from the regional PICU between November 2000 and March 2002. This comparison group included some children who were cared for initially in an adult ICU and were subsequently transferred to the PICU.
The dataset included PIM variables, demographic variables, level of dependency, ICU diagnosis, and outcome. The method for collecting and calculating PIM is published and readily available.4 Levels of dependency are defined by the context of the patient’s level of care: level 1, high dependency; level 2, intensive care associated with intubation and ventilated; level 3, as for level 2 and associated with additional therapeutic procedures such as inotropic support; level 4, as for level 3 and associated with either renal replacement therapy or ECMO (PICS Standards document 2001). All data collection was undertaken by a single dedicated data collector in each hospital, from patient observation charts and notes, and the data were then collated centrally by a regional coordinator. Patient data were anonymised using unique numerical codes, therefore ensuring personal identification information was not available to the study team. The paediatric admissions records of each unit were reviewed at regular intervals to check for the completeness of recruitment. A 10% random sample of data collection forms was checked by the regional coordinator against the original observation charts to ensure accuracy of data collection. Ethical approval for the collection of these data had previously been sought and obtained from all the local research ethics committees (LRECs) of the participating hospitals.5 Therefore specific LREC approval for this analysis was not judged to be necessary.
The characteristics of patients in both groups were compared using the Mann–Whitney test for continuous variables and the χ2 test for categorical variables. The goodness of fit of PIM in this population of patients was tested in two ways. Discrimination evaluates how well PIM distinguishes between patients who lived and those who died, and was assessed using the area under receiver operator characteristic (ROC) plot. Calibration evaluates the performance of PIM at different risks of mortality, and was assessed by examining a Hosmer–Lemeshow 2×10 table. This displays how well the model matches observed outcomes in deciles of the population ranked by probability of death. The mean probability of death and the ratio of observed to expected death rates (standardised mortality ratio) was also calculated.
Data were collected on 374 children admitted to adult ICUs and from 850 admitted to the PICU. Table 1⇓ shows the patient characteristics.
Table 2⇓ shows the sample period, standardised mortality ratios, and the discrimination of the model in adult ICUs and in the regional PICU. The area under the ROC curve was 0.96 (95% confidence interval, 0.93 to 0.99) for the district general hospital adult ICUs, and 0.78 (0.71 to 0.85) for the regional PICU.
Table 3⇓ shows calibration of the PIM model in adult ICUs. There was no significant heterogeneity between observed and predicted deaths across the 10 deciles of risk (χ2 = 4.55 (8 df), p = 0.8). The calibration of PIM in the regional PICU was less good (χ2 = 14.81 (8df), p = 0.06).
In the South West Region of the UK, despite the centralisation of paediatric intensive care, many critically ill children continue to be managed in district general hospital adult ICUs. Such units only treat level 2 patients within an agreed protocol with the lead PICU, based upon a 24 hour rule—that is, children are only admitted to these units if at the outset it is not envisaged that they will require intubation for more than 24 hours. It is important for commissioners and health planners to monitor the activity of such units, and to do this some adjustment for the severity of illness at admission is necessary, using a risk adjustment tool such as PIM.
Analyses of the patient characteristics show significant differences between the two groups. Although this study was not designed to compare outcome, the standardised mortality ratio (SMR) of children managed in the PICU was lower than in adult ICUs, and previous studies of critically ill children have shown that those in the highest risk stratum have a significantly greater risk of dying if treated outside a designated PICU.5 Interestingly, PIM discriminated better between death and survival in district general hospitals than in the regional PICU. One might speculate that the reason for this closer modelling is because the physiological data necessary for PIM are collected at an earlier time point in the child’s hospital course than when the PICU team collects them. Hence it represents primary organ failure, and is less affected by the treatment that may have been incurred before the child was retrieved by a PICU team. Unfortunately, owing to the small number of patients involved in the study it was not possible to evaluate the performance of PIM in different diagnostic groups. Such evaluation is important and in the past it has shown that in PICU patients PIM does not perform well in certain diagnostic groups. If PIM overpredicts or underpredicts death in a specific diagnostic group, then the overall performance of units as assessed by PIM would be influenced by the proportion of patients admitted in this category and bias the estimation of the SMR. Owing to the diagnostic disparity of the treatment groups acting as a potential source of confounding, it would be important to repeat this study with larger numbers of patients in order to assess the calibration of PIM/PIM2 in adult ICUs across diagnostic groups.
In summary, this study shows that PIM performs well as a risk adjustment method in children whose entire care remains in the adult ICU of a district general hospital. PIM discriminated well between death and survival (Az ROC = 0.96 (95% confidence interval, 0.93 to 0.99)) and calibrated well across deciles of risk (goodness of fit χ2 = 4.55 (8 df), p = 0.8). Therefore, on the basis of these findings the performance of PIM can be judged to be satisfactory as a risk adjustment method in this population of patients. However, future attempts by PICAnet to monitor outcome of children cared for on adult ICUs using revised risk adjustment tools such as PIM2 will require further evaluation in this population of patients.