Elsevier

The Lancet

Volume 365, Issue 9453, 1 January 2005, Pages 82-93
The Lancet

Series
External validity of randomised controlled trials: “To whom do the results of this trial apply?”

https://doi.org/10.1016/S0140-6736(04)17670-8Get rights and content

Summary

In making treatment decisions, doctors and patients must take into account relevant randomised controlled trials (RCTs) and systematic reviews. Relevance depends on external validity (or generalisability)—ie, whether the results can be reasonably applied to a definable group of patients in a particular clinical setting in routine practice. There is concern among clinicians that external validity is often poor, particularly for some pharmaceutical industry trials, a perception that has led to underuse of treatments that are effective. Yet researchers, funding agencies, ethics committees, the pharmaceutical industry, medical journals, and governmental regulators alike all neglect external validity, leaving clinicians to make judgments. However, reporting of the determinants of external validity in trial publications and systematic reviews is usually inadequate. This review discusses those determinants, presents a checklist for clinicians, and makes recommendations for greater consideration of external validity in the design and reporting of RCTs.

Section snippets

Limits on external validity

RCTs and systematic reviews are the most reliable methods of determining moderate treatment effects, but external validity is inevitably less than perfect, at least in theory, because the aim is not to measure the benefit that will be derived from treatment in clinical practice. The response to and/or compliance with a treatment can be influenced strongly by the doctor-patient relationship,51, 52, 53 placebo effects,54, 55 and patient preference.56, 57, 58 Yet trialists rightly try where

Trial setting

There is often concern about the generalisability of trials done in secondary or tertiary care to practice in primary care,26, 27, 28, 29 but there are several other ways in which the setting of an RCT can affect external validity.

Selection of patients

Only a small proportion of patients with a specific disorder participate in a particular trial. For example, only about one in every 200–300 patients undergoing carotid endarterectomy in North America at the time got into the large, multicentre RCTs,72 and similar proportions have been reported in breast cancer trials.73 These low rates mean that trials take many years to recruit but are not a problem for external validity as long as the patients randomised in the participating centres are

Characteristics of randomised patients

Trial reports usually include the baseline clinical characteristics of randomised patients and so it is argued that clinicians can assess external validity by comparison with their patient(s).49, 50 This theory is clearly sensible, but baseline clinical characteristics can be misleading. The difficulty in extrapolating from baseline clinical characteristics is illustrated by the patients who were randomised to endarterectomy in the ECST60 but who did not have surgery because their surgeon

The intervention, control treatment, and pre-trial or non-trial management

External validity can also be affected if trials have protocols that differ from usual clinical practice. For example, before randomisation in the RCTs of endarterectomy for symptomatic carotid stenosis patients had to be diagnosed by a neurologist and have conventional arterial angiography,109 neither of which are routine in many centres. The trial intervention itself may also differ from that used in current practice, such as in the formulation and bioavailability of a drug, or the type of

Outcome measures and follow-up

The external validity of an RCT also depends on whether the outcomes were clinically relevant. This can depend on subtle considerations such as who actually measured the outcome, as is illustrated by the lower operative risks of endarterectomy in studies in which patients were assessed by surgeons rather than by neurologists,111 but is more often dependent on what was measured and when.

Adverse effects of treatment

Reporting of adverse effects of treatment in RCTs and systematic reviews is often poor. In a review of 192 pharmaceutical trials, less then a third had adequate reporting of adverse clinical events or laboratory toxicological findings.135 Treatment discontinuation rates provide some guide to tolerability but pharmaceutical trials often use eligibility criteria and run–in periods to exclude patients who might be prone to adverse effects. Rates of discontinuation of treatment are therefore

Validity of routinely collected data

Non-randomised treatment comparisons based on routinely collected data are often suggested to be more externally valid than RCTs because they include all patients, are done in the real world, and include the effects of the doctor-patient relationship and patient preference.144, 145 Although with appropriate adjustment for differences in case-mix the results are sometimes similar to those from RCTs,146, 147 it is impossible to be certain that bias will be evident or amenable to correction. The

Summary and recommendations

Randomised trials and systematic reviews provide the most reliable data on the effects of treatment and many serious errors have resulted from relying on other types of evidence (panel 3). However, although dogmatic refusal by clinicians to accept the results of RCTs is unacceptable, there is justifiable concern that external validity is often poor. This perception is leading to the under-use in routine practice of treatments that have been shown to be effective in trials. Some trials have very

References (149)

  • C Maynard et al.

    Underutilisation of thrombolytic therapy in eligible women with acute myocardial infarction

    Am J Cardiol

    (1991)
  • MM Rovers et al.

    Generalisability of clinical trials in otitis media with effusion

    Int J Pediatr Otorhinolaryngol

    (2001)
  • M Bjorn et al.

    Consecutive screening and enrolment in clinical trials: the way to representative patient samples?

    J Card Fail

    (1998)
  • RB Haynes et al.

    Patient compliance and the conduct and interpretation of therapeutic trials

    Control Clin Trials

    (1987)
  • O Gomez-Marin et al.

    The sodium-potassium blood pressure trial in children: design, recruitment and randomisation

    Control Clin Trials

    (1991)
  • AL Cochrane

    Effectiveness and Efficiency: random reflections on Health Services

    (1972)
  • R Horton

    Common sense and figures: the rhetoric of validity in medicine (Bradford Hill Memorial Lecture 1999)

    Stat Med

    (2000)
  • SJ Pocock

    Clinical trials: a practical approach

    (1983)
  • LM Friedman et al.

    Fundamentals of clinical trials

    (1998)
  • D Black

    The limitations of evidence

    J R Coll Physicians Lond

    (1998)
  • JR Hampton

    Size isn't everything

    Stat Med

    (2002)
  • LR Caplan

    Evidence based medicine: concerns of a clinical neurologist

    J Neurol Neurosurg Psychiatry

    (2001)
  • JG Evans

    Evidence-based and evidence-biased medicine

    Age Ageing

    (1995)
  • BG Charlton et al.

    The rise and fall of EBM

    Q J Med

    (1998)
  • JD Swales

    Evidence-based medicine and hypertension

    J Hypertens

    (1999)
  • MorganWKC

    On evidence, embellishment and efficacy

    J Eval Clin Pract

    (1997)
  • FB Garfield et al.

    Clinical judgement and clinical practice guidelines. Int J Technol

    Assess Health Care

    (2000)
  • MB Cabana et al.

    Why don't clinicians follow clinical practice guidelines? A framework for improvement

    JAMA

    (1999)
  • DA Davis et al.

    Translating guidelines into practice, a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines

    Can Med Assoc J

    (1997)
  • LG Ford et al.

    Effect of patient management guidelines on physician practice patterns: the community hospital oncology program experience

    J Clin Oncol

    (1987)
  • R Grol et al.

    Attributes of clinical guidelines that influence use of guidelines in general practice: observational study

    BMJ

    (1998)
  • FH Messerli

    Antihypertensive therapy: β-blockers and diuretics - why do physicians not always follow guidelines?

    Proceedings/Baylor University Medical Center

    (2000)
  • J Sonis et al.

    Applicability of clinical trial results to primary care

    J Am Med Soc

    (1998)
  • S Wilson et al.

    Randomized controlled trials in primary care: case study

    BMJ

    (2000)
  • J Sellors et al.

    Recruiting family physicians and patients for a clinical trial: lessons learned

    Fam Pract

    (2002)
  • N Oswald et al.

    Applying research evidence to individuals in primary care: a study using non-rheumatic atrial fibrillation

    Fam Pract

    (1999)
  • T Fahey

    Applying the results of clinical trials to patients in general practice: perceived problems, strengths, assumptions, and challenges for the future

    Br J Gen Pract

    (1998)
  • LD Jacobson et al.

    Evidence based medicine and general practice

    Br J Gen Pract

    (1997)
  • JS McCormick

    The place of judgement in medicine

    Br J Gen Pract

    (1994)
  • I Benech et al.

    Evidence-based practice in primary care: past, present and future

    J Eval Clin Prac

    (1996)
  • A Britton et al.

    Threats to applicability of randomised trials: exclusions and selective participation

    J Health Serv Res Policy

    (1999)
  • SH Downs et al.

    The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and nonrandomised studies of health care interventions

    J Epidemiol Community Health

    (1998)
  • S Gilbody et al.

    Randomized controlled trials in schizophrenia: a critical perspective on the literature

    Acta Psychiatr Scand

    (2002)
  • RW Licht et al.

    Generalisability of results of randomised drug trials

    Br J Psychiatry

    (1997)
  • AJ Camm

    Clinical trials of arrhythmia management: methods or madness

    Control Clin Trials

    (1996)
  • SL Norris et al.

    Effectiveness of self-management training in type 2 diabetes: a systematic review of randomised controlled trials

    Diabetes Care

    (2001)
  • MooreDAJ et al.

    How generalisable are the results of large randomised controlled trials of antiretroviral therapy

    HIV Med

    (2000)
  • N Brown et al.

    Relevance of clinical trial results in myocardial infarction to medical practice: comparison of four year outcome in participants of a thrombolytic trial, patients receiving routine thrombolysis, and those deemed ineligible for thrombolysis

    Heart

    (1999)
  • ME Charleson et al.

    Applying results of randomised trials to clinical practice: impact of losses before randomisation

    BMJ

    (1984)
  • GE Simon et al.

    Initial antidepressant choice in primary care: effectiveness of fluoxetine vs. tricyclic antidepressants

    JAMA

    (1996)
  • Cited by (0)

    View full text