Systematic Reviews: Reporting, updating, and correcting systematic reviews of the effects of health care
BMJ 1994; 309 doi: https://doi.org/10.1136/bmj.309.6958.862 (Published 01 October 1994) Cite this as: BMJ 1994;309:862
- I Chalmers,
- B Haynes
- UK Cochrane Centre, NHS R Programme, Oxford OX2 7LG Canadian Cochrane Centre, Health Information Research Unit, McMaster University Medical Centre, Hamilton, Ontario L8N 3Z5, Canada.
The recent growth in the numbers of published systematic reviews reflects growing recognition of their importance for improving knowledge about the effects of health care. In Britain the NHS R'D Programme has established two centres to prepare systematic reviews of existing information, and the Cochrane Collaboration - an international network of individuals and institutions - evolved to produce systematic, periodically updated reviews of randomised controlled trials. The large amount of existing evidence that needs to be considered creates a problem for the reporting of systematic reviews: the need to ensure that methods and results of systematic reviews are adequately described has to be reconciled with the limited space available in printed journals. A possible solution is the use of electronic publications: reviews could be published simultaneously in a short, printed from and in a more detailed electronic form. Electronic publications also have the advantage of the ease with which reviews may be updated as new evidence becomes available or mistakes are identified.
Primary and secondary research on the effects of health care: the dangerous consequences of double standards
It was not until very recently that anyone drew attention to the fact that clinical investigators usually jettison scientific principles when they move from primary research to secondary research (reviews). Mulrow, in 1987, who first showed that this double standard was manifest in some of the world's leading medical journals,1 and Huth, in an accompanying editorial in Annals of Internal Medicine, said that something ought to be done about it.2 The following year Oxman and Guyatt published guidelines to help people to judge the scientific quality and trustworthiness of reviews.3
The failure of clinical investigators to apply scientific principles to control biases and imprecision in their reviews of evidence about the effects of care can have serious consequences. For example, the second edition of the Oxford Textbook of Medicine advised its tens of thousands of readers that “The clinical benefits of thrombolysis [in treating patients with myocardial infarction] whether expressed as improved patient survival or preservation of left ventricular function, remain to be established.”4 This unsupported view appeared four years after Yusuf and his colleagues had shown in a systematic review of the relevant randomised controlled trials that this treatment reduced the risk of premature death after myocardial infarction.5,6 Indeed, as was shown subsequently by Antman and his colleagues, strong evidence in support of thrombolysis would have emerged a decade earlier had a systematic review been conducted then.7
This is just one of many examples that could be used to illustrate the importance of systematic and timely reviews of evidence about the effects of health care. When the research community sysnthesises existing evidence thoroughly, it is certain that a substantial proportion of current notions about the effects of health care will be changed. Forms of care currently believed to be ineffective will be shown to be effective; forms of care thought to be useful will be exposed as either useless or harmful; and the justification for uncertainty about the effects of many other forms of care will be made explicit. In addition, systematic reviews of existing evidence will reveal that many proposals for new research are misguided because they have not taken proper account of available information.
Systematic reviews of existing evidence, using metaanalysis when appropriate and possible, are examples of “advanced clinical research.”8 Because reviews have such an important place in the chain linking basic research and improved human health, the science of reviewing research must be recognised more explicitly, both within the academic community and more widely.
There is encouraging evidence that this proposition is beginning to be accepted in several places. Using the term meta-analysis as a marker for tracking growth in interest in systematic methods of review, a Medline search before 1982 could be expected to yield about one systematic review a year. Between 1982 and 1985 the average annual yield was about 15.9 Since 1986, however, the number has increased dramatically, and a Mediline search using the MeSH term META-ANALYSIS and the text word “meta-analysis” yielded over 500 citations published in 1992 (C Lefebvre, personal communication). The National Library of Medicine acknowledged this advance in 1993 when the term was given the status of a publication type.
This growth in the numbers of systematic reviews certainly reflects both growing recognition of their importance within academia and support from the organisations that employ clinical investigators. Support for systematic reviews has come, in addition, from those who are trying to assemble information that will help to make more effective use of limited resources for health care. For example, in the United States the agency for Health Care Policy and Research - part of the Public Health Service - has invested substantially in reviews of existing evidence about the effects of care, and in the United Kingdom, the NHS R'D Programme has established two centres (in Oxford and York) to help to prepare systematic reviews of existing information. At an international level, a network of individuals and institutions - the Cochrane Collaboration - has evolved in response to Cochrane's criticism of the health professions for not having organised systematic, periodically updated reviews of all relevant randomised controlled trails.10
Cochrane reviews
Publication of Cochrane reviews began in 1993 with the release of a specialised database11 compiled using subsets of reviews contained in the “parent” Cochrane Database of Systematic Reviews. Publication of the Cochrane Database of Systematic Reviews itself, although it will initially contain only a small number of reviews covering a relatively narrow range of topics, will begin at the end of 1994, when it will be issued on CD-ROM.
The magnitude of the task of finding out what can be known from existing evidence about the effects of health care should not be underestimated: it seems likely that, even if attention were to be focused on randomised controlled trials alone, as many as a million studies conducted during the second half of the 20th century may need to be considered. Hundreds of people are already contributing to the Cochrane collaboration and getting to grips with Archie Cochrane's daunting agenda, but it is likely that it will take at least a couple of decades for a stable state to be reached such that the results of new primary research are being incorporated efficiently into an existing body of systematic reviews of previous research.
At least one commentator has pointed out that a case could be made for a moratorium on proposals for additional primary research until existing results of research have been incorporated in scientifically defensible reviews.12 This view is reflected in the fact that some funding bodies have begun to make it clear to potential applicants that they will expect applications for support for new research to be accompanied by systematic reviews of relevant existing evidence. These systematic reviews must show that the proposed new research is necessary and that it has been designed appropriately - in brief, that it is likely to constitute a sensible use of the limited resources available for research.
Reporting systematic reviews of the effects of health care
Many journal editors are responding to suggestions about how they can serve the needs of their readers more effectively,13,14 and one of the ways that they are doing so is by accepting the need to improve the scientific quality of the reviews that they publish.2 This trend has had consequences for reports of primary research as well as for “stand alone” reviews because there is an increasing expectation among readers that investigators will set the results of new primary research in the context of systematic reviews of relevant existing evidence.
This trend has meant that reports of reviews, whether they stand alone or are components of the discussion sections of reports of new primary research, now tend to take up more space than previously. When the totality of relevant evidence requiring review is small, this does not present any great problems.15 When, as is increasingly the case, there is a large amount of existing evidence, journal editors are confronted with a dilemma. On the one hand, they need to ensure that the materials, methods, and results of systematic reviews are reported in sufficient detail to allow readers to assess their scientific quality; on the other hand, these scientific requirements have to be reconciled with constraints imposed by the inevitably limited space available in printed journals. The 29 page, two part Lancet report of the Early Breast Cancer Trialists' Collaborative Group, and the 46 page, three part BMJ report of the Antiplatelet Trialists' Collaboration, are recent examples of the opportunities and problems confronting editors.16,17
The editorial dilemma was made explicit soon after the dawn of the new era of systematic reviews of randomised controlled trials. The 10 page report in the Lancet of the ISIS-1 trial included a discussion section incorporating a systematic review of all the trials of Beta blockade for myocardial infarction.18 This provided an up to date assessment of the available evidence and showed how the results of the new trial contributed to the overall picture. While acknowledging that there was a good case for presenting such analyses in the discussion sections of reports of primary research, an accompanying editorial warned that the Lancet would lead the opposition to anyone suggesting that such “lengthy tailpieces” should become a regular feature of clinical trial reports.19
Electronic opportunities
As it happens, it has been the Lancet (among the major general medical journals) that has been most active in exploring how best to resolve these dilemmas. The Lancet published correspondence in response to its 1986 editorial and subsequently which drew attention to the potential for exploiting simultaneous electronic and paper publication of lengthy systematic reviews.20,21 Within a month of the launch of the first electronically published general medical journal - the Online Journal of Current Clinical Trials - in 1992, the Lancet announced that it had come to an arrangement with the new journal whereby certain reports, including systematic reviews of trials, would be published simultaneously both in short, printed forms and in more detailed electronic forms.22 The first examples of this arrangement for parallel publication of detailed and brief reports of systematic reviews were published the following year.23,24
The principle of concurrent electronic and paper publication of systematic reviews has been reflected in the arrangements agreed by the Cochrane Collaboration with the BMJ and the Lancet, and subsequently with other journals, such as Annals of Internal Medicine. The International Committee of Medical Journal Editors (the Vancouver Group) has also supported the arrangement. The detailed, highly structured reports of systematic reviews prepared for dissemination electronically in the Cochrane Database of Systematic Reviews are eligible for submission to any print journals endorsing these arrangements. If, after assessment, a print journal accepts a Cochrane review for publication, it will be shortened and modified to reflect the style and other requirements of the journal concerned. This shortened and modified version of the review will then be published concurrently with the electronic dissemination of a longer, more structured version through the Cochrane Database of Systematic Reviews.
Full success in reporting the findings of systematic reviews will not be achieved by these means alone.25,26 Improvements are needed in the integration of information from reviews in the development and reporting of clinical guidelines and health policy. There is also a need for innovation in reporting of review findings in medical textbooks, in materials for the continuing education of health professionals, patients, and the public, and in computerised clinical decision supporting systems.27 Some of these innovations have already been piloted in the preparation of a popular guide to the detailed Cochrane reviews of care in pregnancy and childbirth,28 and in the development of clinical and policy guidelines based on them. As the number and variety of systematic reviews grows there will be an increasing need for refined indexing to permit easy retrieval and for organising the reviews in ways that allow them to be easily rearranged to meet particular needs.
Updating and correcting systematic reviews of the effects of health care
A combination of electronic and paper publication will help people to cope with the sheer size of many reports of systematic reviews. Electronic media come into their own, however, when systematic reviews must be updated or corrected as new evidence becomes available and mistakes are identified.19 It is often frustrating for authors (and others) that the printed reports of their reviews cannot be amended when omissions and mistakes are discovered and drawn to their attention. Because these printed reports will often be reproduced in offprints and photocopies and bound in books and library collections, it is inevitable that many readers will be misled, often over a period of many years. Diligent readers wishing to be properly informed must assemble the original reports, together with the correction notices, letters to the editor, and other criticisms and responses published subsequently, and then try to prepare an effective synthesis of all this material. These are difficult and tedious take, particularly when the criticisms and comments are widely scattered.
It is essential that more efficient arrangements are developed for criticising and amending reviews in the light of new evidence and valid criticisms. To achieve this, the Cochrane Collaboration's working methods include a commitment to timely updating and concurrent reporting of criticisms and other responses. This commitment is backed up by the cover sheet for each review contained in the Cochrane Database of Systematic Reviews, which gives the names, addresses, and other contact details (telephone, fax, and electronic mail) both of the principal reviewer and of the editorial team responsible for coordinating the collaborative review group to which he or she belongs.
These requirements of those contributing to the Cochrane Collaboration, taken together with the practical experience acquired by a group of reviewers preparing and maintaining systematic reviews of controlled trials in pregnancy and childbirth,29,30 lie behind the Cochrane Collaboration's adoption of electronic media as a primary means of assembling and disseminating Cochrane reviews.31 As it evolves, the collaboration intends to create an iterative system through which successive versions of each review will reflect not only the emergence of new data but also valid criticisms, solicited or unsolicited, from whatever source.30
An interactive system
The use of successive issues on floppy disk and CD- ROM to update and amend Cochrane reviews in the light of new evidence and criticisms undoubtedly represents an advance beyond the constraints imposed by publishing reviews in print. It will be important to go beyond these arrangements, however, to develop more efficient and transparent mechanisms for maximising the reliability of Cochrane reviews. Efficient online access to the Cochrane Database of Systematic Reviews would seem to provide the most satisfactory basis for this. It should be possible for people consulting the database to append their criticisms and comments to the Cochrane reviews, for the attention not only of Cochrane reviewers and editors but also the readers of the reviews. Although those consulting the Cochrane database would be offered the most recent version of a review as the default option, earlier versions of each review, together with any intervening criticisms, would be archived electronically for consultation if desired. Complementary arrangements will be needed to ensure that other publication forms that use Cochrane reviews are aware of substantive updates.
Although an interactive system of the kind outlined above may take some years to develop, the scientific dividends could be substantial. It should facilitate the critical dialogues on which advances in knowledge so often depend. There is certainly no room for complacency about the existing arrangements for critical assessment of material submitted to and published in scientific journals.32 Electronic publishing can be exploited to extend this critical assessment beyond the handful of individuals selected by journal editors as referees before publication and the handful of correspondents offered space in correspondence columns after publication.
Prospects and challenges
Systematic reviews of research evidence will play an increasing role in the evolution of health services, the design and justification of controlled trials in health care, and the education of health professionals and lay people. The Cochrane Collaboration is helping to promote the development of systematic reviews by setting explicit standards for reviews; by providing a framework within which people can collaborate in preparing and maintaining reviews in areas of mutual interest; by helping to mobilise resources of various kinds for reviewers; and by developing better means for disseminating systematic reviews to all those who may find them helpful. Medical journals and publishers are playing their part through featuring systematic reviews and by fostering favourable copyright arrangements.
Substantial challenges remain, however. Relatively few health care problems have been covered by systematic reviews so far, and access to existing reviews is limited. The demand for systematic reviews vastly exceeds the capacity of those who are prepared to commit themselves not only to preparing reviews which meet acceptable scientific standards but to the long term maintenance of these reviews as new evidence and criticisms emerge. Funding agencies and academic institutions are only just beginning to treat systematic reviews as scientific projects in their own right: for comparable effort, the current rewards for people engaged in primary research remain significantly better than those for people preparing systematic reviews. Traditional textbooks based on opinion and unsystematic reviews continue to be published in vast numbers.
These problems will be overcome as the value of systematic reviews becomes more widely known and as the infrastructure for development, organisation, and dissemination of reviews becomes better established. It will take a concerted effort over many years to reach the point at which existing evidence about the effects of health care has been organised systematically and made readily available to the variety of people who need this information to help them take better decisions in health care and research, but there is every reason to believe that the effort required will be seen to have been worthwhile.
Iain Chalmers is supported by the NHS R'D Programme; Brian Haynes is supported by the Canadian National Research and Development Programme and McMaster University. The Cochrane Collaboration is an international network of individuals and institutions receiving support from a wide variety of sources. The steering group of the Cochrane Collaboration is chaired by Professor David Sackett, formerly at McMaster University, Canada, now at the Nuffield Department of Medicine, University of Oxford.