Article Text

PDF

Calibration of the paediatric index of mortality in UK paediatric intensive care units

Abstract

AIM To test a paediatric intensive care mortality prediction model for UK use.

METHOD Prospective collection of data from consecutive admissions to five UK paediatric intensive care units (PICUs), representing a broad cross section of paediatric intensive care activity. A total of 7253 admissions were analysed using tests of the discrimination and calibration of the logistic regression equation.

RESULTS The model discriminated and calibrated well. The area under the ROC plot was 0.84 (95% CI 0.819 to 0.853). The standardised mortality ratio was 0.87 (95% CI 0.81 to 0.94). There was remarkable concordance in the performance of the paediatric index of mortality (PIM) within each PICU, and in the performance of the PICUs as assessed by PIM. Variation in the proportion of admissions that were ventilated or transported from another hospital did not affect the results.

CONCLUSION We recommend that UK PICUs use PIM for their routine audit needs. PIM is not affected by the standard of therapy after admission to PICU, the information needed to calculate PIM is easy to collect, and the model is free.

  • mortality
  • intensive care
  • prediction model

Statistics from Altmetric.com

Those wishing to audit paediatric intensive care units (PICUs) have to adjust or control their results for variations in “case mix”. In its broadest sense “case mix” describes all the confounding factors that could introduce bias, or the chances of a particular outcome in a comparison between different units or periods of time. The most significant element of case mix is the severity of the illness leading to PICU admission. Traditionally this is mathematically represented as a mortality risk (odds of dying), or the related “probability of death” (POD), as determined by a mortality prediction model derived by logistic regression1 from a large database of consecutive representative admissions. The process of routine audit requires the use of a validated “off the shelf” model, although detailed comparisons for formal epidemiological studies are better done using a study specific model. There is currently a choice between two risk adjustment tools for routine use in children in intensive care: the paediatric risk of mortality (PRISM—currently marketed in its first revision, PRISM III),2 3 and the paediatric index of mortality (PIM).4 Neither model has been adequately evaluated in the UK. We sought to test PIM in five UK paediatric intensive care units.

Methods

The variables that contribute to PIM were collected prospectively in the PICUs at five hospitals: Alder Hey Children's Hospital in Liverpool, Birmingham Children's Hospital, and Great Ormond Street, Guy's, and St George's Hospitals in London. Data collection was by junior medical staff and validated by a named consultant collaborating in the study. Four of the five centres were recognised “lead centres” for paediatric intensive care provision and training, admitting in excess of 800 admissions per year (or more than 600 intubated admissions per year) and catering for a broad cross section of paediatric intensive care diagnostic case mix. The fifth had a smaller throughput and a greater proportion of high dependency cases. However, all were staffed by full time consultant paediatric intensive care physicians and dedicated junior staff with no commitments other than the PICU.

Admissions were classified by diagnostic groups defined in advance as respiratory, cardiac, neonatal, postoperative, accident/trauma, neurological, or miscellaneous.

DATA AND STATISTICS

The data needed to calculate PIM were collected within one hour from the time of first face to face contact between the ICU doctor and the patient, even when this occurred in another hospital—for example, when patients were transported from another institution. The method for calculating the probability of death is detailed in the . In addition, various observational measures of case mix were recorded, such as whether the patient's intensive care admission was a result of interhospital transport and whether or not the patient was intubated during their admission. Patients who were intubated at any point can be regarded as having received a period of intensive care.

We tested the fit of PIM mortality prediction model in two ways. Discrimination was assessed using the area under the receiver operating characteristic (ROC) plot.5 This measure expresses how well the model distinguishes between patients who lived and those who died. An area under the ROC of 0.75 or more is considered clinically useful, and in the design of mortality prediction there is a trade off between the simplicity of the model (easier data collection and quality) and enhanced area under the ROC plot. An area under the ROC plot of 0.75 means that a randomly selected non-survivor would have a higher PIM value than a randomly selected survivor 75% of the time; it does not mean that prediction of death is correct 75% of the time. Calibration evaluates how well the model classifies patients into low, medium, and high risk categories. We evaluated this by examining a Hosmer–Lemeshow goodness of fit 4×10 table, which displays how well the model matches observed outcomes in deciles of the population ranked by probability of death.1 The results are also presented as observed:expected mortality ratios within standard risk categories (predicted mortality <1%, 1–4%, 5–14%, 15–29%, or 30% or more) and diagnostic categories. The overall expected death rate is the sum of the probability of death for each admission, and the ratio of observed to expected death rates is known as the standardised mortality ratio (SMR). Values less than one imply good performance, and values greater than one imply poor performance, although either may be affected by the fit of the model (see below). Confidence limits for the SMR were derived from a parametric approach.6

Results

We analysed a total of 7253 admissions. One centre (Great Ormond Street) did not collect data on consecutive admissions in all its intensive care units, so these were excluded. However, data from the cardiac ICU were complete for a six month period and were included. Table 1 shows the sample periods, standardised mortality ratios, number of patients, and the discrimination of the model in each hospital. The area under the ROC plot was 0.84 (95% confidence interval (CI) 0.819 to 0.853; see fig 1). Table 2 shows calibration of the PIM model. Table3 shows the performance of the model in different diagnostic categories and in the standard severity of illness bands.

Table 1

Data by hospital

Table 2

Calibration of the PIM model

Table 3

Calibration across diagnostic categories and severity of illness bands

There were clinically significant differences in the proportion of patients transferred by an intensive care transport team to the receiving hospital, which were not associated with the proportion that were ventilated at admission to PICU (r = −0.05). There was remarkable similarity in the performance of the PICUs as assessed by the SMR. The overall SMR was 0.87 (95% CI 0.81 to 0.94) and this apparent good performance of the units appears on calibration to be systematic (table2). As expected for a tool derived from a population of all intensive care admissions, the performance of the model was better in some diagnostic groups than in others. The most common diagnoses in the miscellaneous group were septic shock (204 children), other infection (71), liver transplant (64), neoplasm (53), acute hepatitis or liver failure (49), haemopoietic disease (48), renal failure (38), and inborn errors of metabolism (31).

Discussion

Until now, neither PIM nor PRISM has been adequately tested in UK paediatric intensive care units, and the performance of PRISM has been questioned.7 8 Mortality prediction models which, like PRISM, use the worst values of their predictor variables in the first 12–24 hours in PICU, have three disadvantages.9 Firstly, the data are difficult to collect. Secondly, they are not a good tool for comparing different intensive care units: for example, patients mismanaged in a poor quality unit will have higher scores than similar patients managed in a high quality unit, so the high mortality rate of the former might be incorrectly attributed to its having sicker patients. Thirdly, they appear to be more accurate than they really are: about 40% of deaths occur in the first 24 hours in PICU, so in these cases the score is really diagnosing death rather than predicting it.

PIM is free, and the data are collected within an hour of the time of first face to face contact between a PICU doctor and the patient. This means that the data have to be recorded by the PICU doctor rather than a data clerk, but it avoids the problem, common to all 12–24 hour scores, that the standard of care provided in a unit alters the predicted mortality rate. First contact data collection also means that PICU retrieval is, justifiably, evaluated as part of the PICU care. PIM also adjusts for the influence of premorbid states that can profoundly influence outcome.

PIM compares PICU performance to that of the PICUs that contributed to the derivation dataset. PIM was derived from data that were collected from consecutive admissions to all seven dedicated PICUs in Australia and one in the UK in 1994–95, so it is not surprising that UK PICUs in 1998–99 had 13% fewer deaths than predicted by the model. A revised version of PIM will be available soon to bring the model up to date.

Sixty per cent of the patients and 65% of the deaths in this study were from Birmingham Children's Hospital (table 1). However, this does not invalidate our finding that PIM works well: the SMR and area under the ROC plots were remarkably similar in all five PICUs (table 1).

When an individual PICU or group of PICUs applies a logistic regression model such as PIM to their data, differences between the observed and expected number of deaths may be a result of either the performance of the ICU, or the performance of the model. The subsequent clinical interpretation is rarely objective if there are more deaths than predicted; there is a tendency to blame the fit of the model for the discrepancy. A proliferation of new mortality prediction models results, because the coefficients of the original model are often changed to compensate for poor performance. In fact, consistent differences across deciles of risk, as seen in this study, are more likely to be a result of differences in clinical performance rather than fundamental errors in the structure of the model. The reverse is largely true for non-systematic errors. It is not usually appropriate to respond to an SMR of more than 1.0 (which implies poor clinical performance) by changing the model. Furthermore one must be very cautious in interpreting small series (for example, with fewer than 20 deaths per unit).

CONCLUSION

We found that PIM provides a consistent guide to the performance of these five PICUs. Great care has to be taken to ensure that the data needed to calculate PIM are accurate; the fact that all the information is collected in the first hour simplifies this task. The model incorporates the quality of retrieval services in its assessment, and adjusts for the presence of important premorbid conditions. We recommend that PIM be used routinely as a mortality prediction model for paediatric intensive care in the UK.

Acknowledgments

The collaborators in this study, who collected and cleaned the data from their institutions, were Dr P Baines (Alder Hey), Dr A Goldman (Great Ormond Street), Dr P J Rye (St George's), and Dr I Murdoch (Guy's). Further information about PIM can be obtained in the software section of http://pedsccm.wustl.edu/.

Appendix

PIM is calculated from information collected at the time a child is admitted to your ICU. Because PIM describes how ill the child was at the time you started intensive care, the observations to be recorded are those made at or about the time of first face to face (not telephone) contact between the patient and a doctor from your intensive care unit (or a doctor from a specialist paediatric transport team). Use the first value of each variable measured within the period from the time of first contact to one hour after arrival in your ICU. The first contact may be in your ICU, or your emergency department, or a ward in your own hospital, or in another hospital (e.g. on a retrieval). The pupils' reactions to light are used as an index of brain function; do not record an abnormal finding if this is probably caused by drugs, toxins, or local injury to the eye. If information is missing (e.g. base excess not measured), record zero (except for systolic blood pressure, which should be recorded as 120); PIM assumes that missing values are normal (e.g. that the base excess is 0 if it is not measured). Variables 1–7 are descriptive variables that are not needed to calculate PIM. Include allchildren admitted to your ICU.

(1)
Identifier (e.g. ICU admission number):
(2)
Age (months):
(3)
Diagnostic group (1 = Resp, 2 = CVS, 3 = Postop, 4 = Accident, 5 = Neurol, 6 = Other):
(4)
Date admitted to ICU (dd/mm/yyyy):
(5)
Days in ICU on this admission:
(6)
Endotracheal tube in situ at any time during this ICU admission (no = 0, yes = 1):
(7)
Outcome of ICU admission (discharged from ICU = 0, died in ICU = 1):
(8)
Booked admission to ICU after elective surgery; or elective admission for a procedure (e.g. insertion of a central line), or monitoring, or review of home ventilation (no = 0, yes = 1):
(9)
Record the number in square brackets if the condition is present (if in doubt, record 0): [0] none [1] cardiac arrest out of hospital [2] severe combined immune deficiency [3] malignancy after completion of 1st induction [4] spontaneous cerebral haemorrhage from aneurysm or AV malformation [5] cardiomyopathy or myocarditis [6] hypoplastic left heart, <1 mth, requiring Norwood [7] HIV infection [8] IQ <35, worse than Down's [9] a neurodegenerative disorder (progressive ongoing loss of milestones)
(10)
Response of pupils to bright light (>3 mm and both fixed = 1, other = 0, unknown = 0):
(11)
Base excess in arterial or capillary blood, mmol/l (unknown = 0):
(12)
Pao 2, mm Hg (unknown = 0):
(13)
Fio 2 at time of Pao 2 if oxygen via ETT or headbox (unknown = 0):
(14)
Systolic blood pressure, mm Hg (unknown = 120):
(15)
Mechanical ventilation at any time during the first hour in ICU (no = 0, yes = 1):

example of how to calculate pim

Child has: pupils react (fixed = no = 0), myocarditis (specified diagnosis = yes = 1), emergency admission (elective = no = 0), ventilated (= yes = 1), systolic blood pressure 40 mm Hg, base excess −16.0 mmol/l, Fio 2 1.00, and Pao 2 60 mm Hg.
PIM logit = (2.357 × 0) + (1.826 × 1) + (−1.552 × 0) + (1.342 × 1) + (0.021 × absolute(40 − 120)) + (0.071 × absolute(−16.0)) + (0.415 × 100 × 1.00/60) − 4.873 = 1.803.
Predicted probability of death = elogit / (1 + elogit) = 2.71831.803 / (1 + 2.71831.803) = 0.8585, or 86%.
At <www.pedsccm.wustl.edu/clinical/pim-readme.html> there is a free program that calculates PIM.

common mistakes in collecting pim data

(1)
Do not overdiagnose the specified conditions (number 9 on the form)—if there is any doubt, do not record a specified condition. For example: do not code cerebral haemorrhage for intracerebral bleeding associated with trauma; impaired cardiac function associated with sepsis or surgery should not be coded as cardiomyopathy; Down's syndrome should not be coded as IQ <35; and a static disability should not be coded as neurodegenerative (even if it is severe) unless there is progressive ongoing loss of milestones. “Lymphoma/leukaemia after 1st induction” has been changed to “malignancy after 1st induction”.
(2)
You should record the first value of each variable from the time of first contact up to one hour after arrival in your ICU (not the worst value).
(3)
If a variable is not measured within one hour of admission to ICU it should be coded as missing (for example, if the first blood gas is not done until two hours after admission, the base excess and Pao 2 should both be coded as missing). Missing data are treated as being normal when PIM is calculated.
(4)
The PIM equation is used to calculate the PIM logit. If any information is missing, that variable should add nothing to the PIM logit. For example, if the Pao 2 or the Fio 2 is missing, the value of “0.415 × 100 × Pao 2/Fio 2” should be set to zero.
(5)
Record the Fio 2 being given at the same time that the first Pao 2 is measured (that is, both the Fio 2 and Pao 2 that you record must relate to the same time).
(6)
Make sure that you are consistent about the Pao 2 units (all mm Hg, or all kPa), and the Fio 2 (all between 0.0 and 1.0, not percentages between 0 and 100).
(7)
Read very carefully the definition, “booked [prearranged] admission to ICU after elective surgery; or elective admission for a procedure (e.g. insertion of a central line), or monitoring, or review of home ventilation”.
(8)
The pupils are only recorded as fixed if both are >3 mm, and both are fixed, and the finding is not caused by drugs or toxins or direct injury to the eye.
(9)
If systolic blood pressure is not measured in the first hour, record 120—do not record zero.
(10)
Randomly sample about every 20th admission to your ICU and get another person to collect the PIM data independently a second time, so that you can check the accuracy of your data.
(11)
You should include all admissions to your ICU, not just selected cases.

References

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Archives this month
    BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health
  • Rapid responses
    BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health
  • Rapid responses
    BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health
  • Rapid responses
    BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health