Diagnostic accuracy of clinical decision rules

A systematic review of the literature was undertaken to evaluate the diagnostic performance of clinical decision rules.6 We only included decision rules where data were available to calculate the sensitivity of the rule for neurosurgical and non-neurosurgical lesions separately, and where there were sufficient numbers of neurosurgical lesions to provide a meaningful estimate of sensitivity. Table 1 shows the sensitivity and specificity of each strategy used as probabilities in the model and the decision nodes these apply to.

The decision analysis model

We developed a decision analysis model using Simul8 software (Simul8 Corporation) to estimate the costs and quality-adjusted life-years (QALYs) gained for the following diagnostic strategies: discharge all patients without testing (theoretical), CT scan all patients (theoretical), Children's Head injury Algorithm for the prediction of Important Clinical Events (CHALICE),2 Paediatric Emergency Care Applied Research Network (PECARN),3 University of California-Davis4 (UCD) and the rule of Atabaki et al., 2008.5 The analysis was conducted for patients aged 1 and 10 years at index event. The model takes a lifetime horizon with mean life expectancy based on UK interim life tables, averaging the value for males and females.7 The economic perspective is the UK National Health Service (NHS) in England and Wales. A simplified schematic of the model is presented in figure 1. Each strategy was applied to a hypothetical cohort of 10 000 patients attending the emergency department with isolated, closed minor head injury (ie, with no other serious injuries). A proportion of the cohort was assumed to have an intracranial lesion requiring immediate neurosurgery (an intracranial haematoma requiring evacuation); a further proportion was assumed to have an intracranial lesion that did not initially require immediate neurosurgery. For convenience, we refer to the former patients as having a neurosurgical lesion and the latter as having a non-neurosurgical lesion, although it should be recognised that the latter may ultimately receive neurosurgical intervention. The remainder would have no intracranial haemorrhage. These proportions were estimated from Smits et al (neurosurgical injury 0.53%, 95% CI 0.33% to 0.85%; non-neurosurgical injury 7.1%, 95% CI 6.26% to 8.05%).8

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Model pathways.

We assumed that the decision rules would determine which patients underwent CT scanning and that the probability of detecting a neurosurgical lesion or a non-neurosurgical lesion was determined by the sensitivity of the rules for these lesions. We assumed that patients with a neurosurgical lesion detected on CT would be managed before any deterioration occurred, while those who did not have a CT had treatment after deterioration had occurred. We assumed that a proportion of patients with a non-neurosurgical lesion would deteriorate over the following 48 h and require critical care support and/or neurosurgery, while the remainder would remain well. These proportions were taken from a study by Fabbri et al.9 We assumed that patients without an intracranial lesion remained well and did not deteriorate. The proportion of these patients receiving unnecessary CT was determined by the specificity of each strategy.

Table 1 shows the sensitivity and specificity parameters of each strategy in correctly determining that a CT scan was required, predicated on the assumption that CT scanning has 100% sensitivity and 100% specificity in identifying significant intracranial lesions. Relevant lesions were those related to the head injury (ie, we did not consider incidental findings unrelated to the injury). The strategy of CT for all therefore had 100% sensitivity for both neurosurgical and non-neurosurgical lesions and 100% specificity. The strategy of discharge for all had zero sensitivity and 100% specificity.

Patient outcome

Patient outcome is measured by the Glasgow Outcome Score (GOS), defined as: GOS 1 Dead; GOS 2 Vegetative state; GOS 3 Severe disability; GOS 4 Moderate disability; GOS 5 Good recovery. The model designated each patient to a GOS category according to whether they had an intracranial lesion and how promptly it was treated. This involved estimating the probabilities that patients with neurosurgical and non-neurosurgical lesions would end up in each GOS category, depending upon the extent of treatment delay. For prompt treatment of patients with a neurosurgical injury, we meta-analysed the data from five studies identified from a systematic review of the literature.10–14

For delayed treatment of patients with a neurosurgical injury, we used Bayesian techniques to estimate GOS outcomes using a study by Haselberger et al, which showed the association between outcome and time delay from loss of consciousness to operation.14

For patients with a non-neurosurgical lesion where treatment is timely and appropriate, we based GOS outcomes on a study by Fabbri et al.9

No studies were found that reported the outcomes of delayed treatment on non-neurosurgical lesions; we therefore, assumed that outcomes were similar to those in delayed treatment of neurosurgical lesions and adjusted the GOS outcomes from Fabbri et al accordingly.9 The GOS outcomes can be seen in table 2.

Costs

Costs included were: cost of investigation, CT scanning, hospital admission, neurosurgical treatment, intensive care and long-term nursing and rehabilitation. These were taken from the UK Department of Health National Schedule of Reference Costs 15 and the Personal Social Services Research Unit.16 Patients with outcome GOS 5 did not incur any further costs. The cost of GOS states 3 and 4 were taken from a study by Beecham.17 This study estimated costs for patients aged 18–25, and as cost data for younger people were not available, we assumed that costs are the same as for age 18–25. The above study did not provide cost data for patients in a vegetative state. We have therefore based our estimates on expert opinion (Sophie Duport, Royal Hospital for Neuro Disability, Personal communication. 8 May 2010). The cost of glioma was taken from the only relevant and reliable data source identified.25 Costs used in the model are shown in table 3.

Quality of life utility values

A literature review was conducted to identify studies that estimated health-related utility values for GOS states. Two studies were found.19 ,20 The Smits et al study was considered to comply most closely with the National Institute for Health and Care Excellence (NICE) reference case and utility values from this study were therefore used.19 ,21

The Smits et al study did not report the age distribution of those patients used to estimate quality of life (QoL) utilities.19 We assumed that QoL for GOS 3 and GOS 4 is not age-related. We have also assumed that QoL for GOS 5 is not age-related. The QoL utility values for death and persistent vegetative state were assumed to be 0. Utilities used in the model are shown in table 4.

Costs and utilities were discounted at an annual rate of 3.5% as recommended by the NICE guide to the methods of technology appraisal.21

Radiation risk

The estimated risk of a tumour from a single CT scan is taken from a study by Stein et al, which also provides an estimated QoL decrement risk per CT scan.22 The Stein et al study reports that the latency between radiation exposure and tumour diagnosis is over 5 years in the majority of cases.22 We have assumed a mean latency period of 10 years and the cost of cancer is therefore discounted for this time period. The values estimated by Stein et al and used in the model can be seen in table 4.

Persistent vegetative state

The Multi-Society Task Force on Persistent Vegetative State reported the mean length of survival for children in a vegetative state as 7.4 years.23 Patients in GOS 2 accrued the costs associated with a vegetative state for this length of time and were then assumed to have died.

Definition of cost-effectiveness terms

The incremental cost-effectiveness ratio (ICER) measures the relative value of two strategies and is the difference in mean costs divided by the difference in mean benefits. Where a strategy is less effective and more expensive than its comparator, it is dominated. Extended dominance occurs when a combination of two alternative strategies can produce the same QALYs as a chosen strategy, but at a lower cost.24 Strategies that are neither dominated nor extendedly dominated constitute the cost-effectiveness frontier, and the ICER is reported for these strategies compared with the next least effective strategy.

In a probabilistic sensitivity analysis (PSA), each parameter in the model is assigned a distribution; see tables 2⇑–4. β distributions were used for probabilities as this distribution is bounded between 0 and 1; Gamma distributions were used for costs as the Gamma can describe the potential skewness of costs; Dirichlet distributions were used for GOS states as the states are correlated and must sum to 1; normal distributions were used for all other parameters. Results are presented as mean and incremental costs, QALYs and ICERs. A cost-effectiveness acceptability curve (CEAC) is calculated using the probability that a strategy has the highest net benefit (NB) for each iteration of the model for a range of willingness to pay (WTP) values.25 Net benefit is defined as: NB=WTP*QALY-cost.

Sensitivity analysis

Univariate sensitivity analysis was undertaken to explore the impact of changing each parameter in the model to its lowest CI, and then, highest CI; or if not available by an arbitrary lower value, then by an arbitrary higher value.

Probabilistic sensitivity analysis

PSA was undertaken to explore the impact of the joint uncertainty surrounding all model parameters. For each model iteration (1000 simulations each with 10 000 patients), each model parameter is randomly sampled from the relevant distribution. Monte Carlo sampling techniques were used to produce information on the likelihood that each intervention produces the greatest amount of NB. ICER is reported for strategies that constitute the cost-effectiveness frontier compared with the next least effective strategy.