SeriesExternal validity of randomised controlled trials: “To whom do the results of this trial apply?”
Section snippets
Limits on external validity
RCTs and systematic reviews are the most reliable methods of determining moderate treatment effects, but external validity is inevitably less than perfect, at least in theory, because the aim is not to measure the benefit that will be derived from treatment in clinical practice. The response to and/or compliance with a treatment can be influenced strongly by the doctor-patient relationship,51, 52, 53 placebo effects,54, 55 and patient preference.56, 57, 58 Yet trialists rightly try where
Trial setting
There is often concern about the generalisability of trials done in secondary or tertiary care to practice in primary care,26, 27, 28, 29 but there are several other ways in which the setting of an RCT can affect external validity.
Selection of patients
Only a small proportion of patients with a specific disorder participate in a particular trial. For example, only about one in every 200–300 patients undergoing carotid endarterectomy in North America at the time got into the large, multicentre RCTs,72 and similar proportions have been reported in breast cancer trials.73 These low rates mean that trials take many years to recruit but are not a problem for external validity as long as the patients randomised in the participating centres are
Characteristics of randomised patients
Trial reports usually include the baseline clinical characteristics of randomised patients and so it is argued that clinicians can assess external validity by comparison with their patient(s).49, 50 This theory is clearly sensible, but baseline clinical characteristics can be misleading. The difficulty in extrapolating from baseline clinical characteristics is illustrated by the patients who were randomised to endarterectomy in the ECST60 but who did not have surgery because their surgeon
The intervention, control treatment, and pre-trial or non-trial management
External validity can also be affected if trials have protocols that differ from usual clinical practice. For example, before randomisation in the RCTs of endarterectomy for symptomatic carotid stenosis patients had to be diagnosed by a neurologist and have conventional arterial angiography,109 neither of which are routine in many centres. The trial intervention itself may also differ from that used in current practice, such as in the formulation and bioavailability of a drug, or the type of
Outcome measures and follow-up
The external validity of an RCT also depends on whether the outcomes were clinically relevant. This can depend on subtle considerations such as who actually measured the outcome, as is illustrated by the lower operative risks of endarterectomy in studies in which patients were assessed by surgeons rather than by neurologists,111 but is more often dependent on what was measured and when.
Adverse effects of treatment
Reporting of adverse effects of treatment in RCTs and systematic reviews is often poor. In a review of 192 pharmaceutical trials, less then a third had adequate reporting of adverse clinical events or laboratory toxicological findings.135 Treatment discontinuation rates provide some guide to tolerability but pharmaceutical trials often use eligibility criteria and run–in periods to exclude patients who might be prone to adverse effects. Rates of discontinuation of treatment are therefore
Validity of routinely collected data
Non-randomised treatment comparisons based on routinely collected data are often suggested to be more externally valid than RCTs because they include all patients, are done in the real world, and include the effects of the doctor-patient relationship and patient preference.144, 145 Although with appropriate adjustment for differences in case-mix the results are sometimes similar to those from RCTs,146, 147 it is impossible to be certain that bias will be evident or amenable to correction. The
Summary and recommendations
Randomised trials and systematic reviews provide the most reliable data on the effects of treatment and many serious errors have resulted from relying on other types of evidence (panel 3). However, although dogmatic refusal by clinicians to accept the results of RCTs is unacceptable, there is justifiable concern that external validity is often poor. This perception is leading to the under-use in routine practice of treatments that have been shown to be effective in trials. Some trials have very
References (149)
Grey zones in clinical practice: some limits to evidence-based medicine
Lancet
(1995)- et al.
Problems in the “evidence” of “evidence-based medicine”
Am J Med
(1997) - et al.
Trends in use of drug therapies in patients with acute myocardial infarction: 1988–1992
J Am Coll Cardiol
(1994) Can randomised trials inform clinical decisions about individual patients?
Lancet
(1999)- et al.
The case for better research standards in peripheral thrombolysis: poor quality of randomised trials during the past decade
Acad Radiol
(1996) - et al.
Placebo effect in double-blind clinical trials: a review of interactions with medications
Lancet
(1994) Powerful placebo: the dark side of the randomised controlled trial
Lancet
(1998)- et al.
An assessment of the feasibility and impact of shared decision making in prostate cancer
Urology
(1998) - et al.
Analysis of randomised and non-randomized patients in clinical trials using the comprehensive cohort follow-up study design
Control Clin Trials
(1992) Variation in protection by BCG: implications of and for heterologous immunity
Lancet
(1995)