Article Text
PDF
Statistics from Altmetric.com
The current movement towards evidence-based medical practice involves integrating individual clinical expertise with the explicit and informed use of the best available external clinical evidence from a systematic review of published research, to reach decisions about the care of individual patients.1 The recent creation of a Centre for Evidence-Based Child Health in London, UK illustrates the increasing priority given to these activities within paediatric practice.2 The clinical evidence from published research comprises patient centred clinical research in a number of areas, including the accuracy and precision of diagnostic tests, the power of prognostic markers, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens. Randomised controlled trials (RCTs) are the gold standard for scientific studies in the last category, including those evaluating the effect of treatment. The primary advantage of RCTs is that when treatments are allocated at random in a sample of sufficiently large size, the results are likely to achieve a level of validity that is not possible to attain with other study designs. RCTs are therefore being used increasingly to evaluate the effectiveness of health care interventions.3
Recognising that RCTs provide the best evidence in comparisons of treatment, rehabilitative or preventive regimens, the Cochrane Collaboration has established an international network of centres that systematically attempt to identify, document, and make available for reference all published and unpublished RCTs.4 The impact of this global initiative in improving the outcome of clinical practice will be dependent on the extent to which clinicians change their practice in the light of available medical evidence. The formation in the UK of the Cochrane Collaborative Review Group on Effective Professional Practice, which focuses on reviews of effectiveness of different approaches to implementation of evidence-based clinical guidelines,5 and the NHS centre for the dissemination of reviews are two initiatives that seek to promote the adoption of evidence-based clinical guidelines and appropriate changes in clinical practice.
One of the authors of this report (HC) is registered with the Cochrane Collaboration to coordinate hand searching of theArchives of Disease in Childhood to identify all published RCTs. This review reports on a descriptive study based on hand searching 15 years of the Archives of Disease in Childhood (excluding the Fetal and Neonatal edition) to identify all RCTs. The data on RCT citations have been made available to the Cochrane Collaboration as part of the international effort to identify RCTs within paediatric practice. In addition to this information we used a structured questionnaire to collect additional specific descriptive data from each of the RCTs identified. This review describes the 249 RCTs published inArchives of Disease in Childhood from 1982–96, and presents some conclusions and recommendations based on our review.
Methods
IDENTIFICATION OF RCTs AND INCLUSION CRITERIA
We systematically hand searched all regular issues ofArchives of Disease in Childhood published from January 1982 to December 1996. We did not include theFetal and Neonatal editions throughout this period but anticipate hand searching these in the next phase of the exercise. In each volume every article was read, including all annotations, review articles, and letters to the editor. A standard form obtained from one of the Cochrane Collaborative review groups (on peripheral vascular diseases) was completed for each study that was an RCT (randomisation by computer, pharmacy codes, random numbers, coin crossing, sealed envelope, or by an unspecified random method); a quasi-RCT (randomisation by date of birth, days of the week, case record, or alternation); or a controlled clinical trial in which there was no mention of the word “random” but where the trial had one of the following features—controlled, double blind, placebo controlled, or crossover design. A separate note was made of all reviews or meta-analyses of RCTs published.
DESCRIPTIVE DATA COLLECTED ON EACH RCT
In addition to these data, which were gathered for the purposes of the Cochrane controlled trials register, a number of additional pieces of data were recorded (table 1) in a systematic manner from each RCT or quasi-RCT in an attempt to describe these in detail. These data included the country or region in which the trial was carried out, the primary setting for the trial (hospital inpatient, hospital outpatient, or community setting) and the subspecialty area of paediatric practice involved. We also gathered data on characteristics of the study design including method of allocation to treatment groups, unit of allocation, trial structure (parallel group, crossover, or factorial design), number of children recruited, nature of the primary intervention, and outcome of the RCT. Finally we recorded the funding source.
Additional data gathered on all identified randomised controlled trials (RCTs) or quasi-RCTs
Consolidation of standards for reporting trials–CONSORT
QUALITY ASSESSMENT
Three key parameters of the quality of the RCT design were also assessed against published standards.6 7 Thus all trials were scored according to the following three factors.
How subjects were allocated to the different treatments (control of selection bias on entry)
This considered whether those enroling study participants could know which treatment was next to be assigned.
-
RCTs that used blinding of randomisation assured by telephone communication or indistinguishable drug treatments randomly precoded by a pharmacy were given a score of 3.
-
RCTs using sealed envelopes and therefore creating a small chance of the next treatment being discovered by those enroling participants were given a score of 2.
-
Trials in which there was no attempt at blinding of the treatment allocation—for example, by use of patient record numbers, birth dates, dates of admission, or alternate allocation were given a score of 1.
Extent to which the primary analysis was based on all participants allocated to receive one or other of the alternative treatments (control of selection bias after entry)
-
Trials in which the primary analysis was based on all cases entered were given a score of 3.
-
Trials in which withdrawals were judged to be too few to engender any major bias were given a score of 2.
-
Others were given a score of 1.
Extent to which the comparisons of outcomes in the trial groups was likely to have been affected by knowledge of the treatment allocation (control of bias in assessing outcomes)
-
Trials in which blinding was of little relevance because only unambiguous outcomes (such as deaths) were compared or where blinding was accomplished successfully—for example, placebo controlled RCTs, were given a score of 3.
-
Studies that were only partially successful in achieving blinding were given a score of 2.
-
Trials in which little or no attempt was made to reduce observer bias were given a score of 1.
COMPARISON OF HANDSEARCHING WITH MEDLINE SEARCH STRATEGY
We also considered whether there was an explicit statement of the inclusion and exclusion criteria for entry to the trial, and of the consent rate among eligible participants. We compared the RCTs identified by this exhaustive hand searching process to RCTs and quasi-RCTs identified by the highly sensitive medlinesearch strategy based on the strategy constructed by Carol Lefebvre of the Cochrane Collaboration (table 2), which is used as the basis of medline searching by all Cochrane Collaboration review groups.8 (This strategy is periodically updated and improved—for details of the most up to date version please consult the Cochrane handbook.9)
medline search strategy for identification of randomised controlled trials
ANALYSIS
Structured questionnaires containing the data gathered from each RCT report were coded following a coding schedule then entered into an EpiInfo 6 database (available at websitehttp://www.cdc.gov/epo/epi/downepi6.htm).10 After checks were made for range and logic errors and missing data (original RCT reports were reviewed where necessary to capture required data), frequency tables were produced. To look at trends over time RCTs were grouped into three five-year bands according to their publication date. Because of small numbers in individual areas, RCTs were also grouped by specialty area of paediatrics for some analyses. These areas were general paediatrics (including infectious diseases, allergies, and dermatology), respiratory, neonatal, and other specialties (including cardiology, endocrinology, gastroenterology, haematology, oncology, nephrology, neurology, and surgery and anaesthesia). Respiratory and neonatal paediatrics were the specialty areas with the largest representation among published RCTs.
Results
A total of 249 original reports of RCTs (228) or quasi-RCTs (21) were identified over the 15 year period 1982–96; 83 were published in 1982–85, 73 in 1987–91, and 93 in 1992–96. The number of RCTs in paediatric specialty areas were as follows: respiratory 82 (33%), general paediatrics 69 (28%), neonatal 35 (14%; NBFetal and Neonatal editions were not hand searched), endocrine 23 (9%), gastroenterology 12 (5%), neurology 8 (3%), haematology/oncology 7 (3%), cardiology 5 (2%), nephrology 5 (2%), and surgery/anaesthesia 3 (1%). This distribution changed little over the 15 year period apart from the reduction in the proportion of neonatal RCTs, which accompanied the introduction of a separate Fetal and Neonatal edition of the journal.
All controlled clinical trials (which were not RCTs) were identified and reported to the Cochrane Collaboration; however, they were not considered in this review. There were no meta-analyses of RCTs published. Seventy per cent (174/249) of trials reported a positive outcome and 30% (75/249) a negative outcome of the intervention under investigation. The number of children recruited in total was 0–19 in 49 RCTs, 20–39 in 70 RCTs, 40–59 in 47 RCTs, 60–79 in 19 RCTs, 80–99 in 18 RCTs, 100–199 in 25 RCTs, 200–299 in 9 RCTs, 300–399 in 5 RCTs, 400–499 in 2 RCTs, 500–999 in 3 RCTs, and ⩾ 1000 in 2 RCTs. The median number of children recruited was 110 in RCTs in neurology, 87 in haematology/oncology, 58 in neonatology, 56 in general paediatrics, 29 in respiratory paediatrics, 24 in cardiology and gastroenterology, 20 in surgery/anaesthesia, and 11 in nephrology. Multicentre RCTs recruited a median of 80 children compared to 36 in single centre RCTs. Twenty four per cent of single centre RCTs recruited fewer than 25 children, 54% recruited fewer than 40 children, 73% recruited fewer than 60 children, and 87% recruited fewer than 100 children.
The medline search strategy (table 2) identified 207 (83%) of the RCTs published in the 15 year study period. This proportion varied significantly with year of publication: 63/83 (76%) of RCTs published in 1982–86, 61/73 (84%) in 1987–91, and 83/93 (89%) in 1992–96. The proportion identified also varied significantly with specialty area: 73/82 (94%) respiratory RCTs, 59/69 (86%) in general paediatrics, 23/35 (66%) neonatal RCTs, and 48/63 (76%) RCTs in other specialty areas. Within this last group the medline search strategy identified 17/23 in endocrinology, 10/12 in gastroenterology, 5/8 in neurology, 6/7 in haematology/oncology, 4/5 in cardiology, 4/5 in nephrology, and 2/3 in surgery/anaesthesia. The relatively low capture of neonatal studies in this review is partly confounded by the fact that there are relatively few recent reports of RCTs in this specialty area (following the introduction of the separateFetal and Neonataledition).
Tables 3 and 4 illustrate the type of trial design (crossover, factorial, or parallel groups), the principal setting of the trial (inpatient, outpatient, or community based) and nature of intervention (pharmaceutical only, pharmaceutical plus placebo control, non-pharmaceutical) by paediatric specialty grouping. These show the small number of factorial design RCTs and the scarcity of community based RCTs published in areas outside general paediatrics. Respiratory medicine is the only specialty area in which most trials of pharmaceutical interventions were placebo controlled (47/73 (64%) in respiratory paediatrics v 37/103 (36%) for RCTs in other specialty areas). Pharmaceutical companies gave funding support to 47/84 (56%) of all placebo controlled RCTs compared to 61/165 (37%) of all other RCTs.
Number of randomised controlled trials by trial design and speciality area
Number of randomised controlled trials by nature and setting of the intervention, and by specialty area
Table 5 shows the number of RCTs reported by geographic area. This shows that both the absolute number and proportion of RCTs published in the Archives of Disease in Childhood from European and Asian centres have increased whereas the numbers from North America, Australia, and Africa have decreased over the 15 year period.
Geographical distribution of randomised controlled trials published 1982–96
Many of the published reports gave insufficient details to be able to assess key quality features of trial design. Reporting of details for blinding of outcome assessment (published in 94% of reports) and completeness of data in the primary analysis (in 100% of reports) were excellent, but fewer than half (43%) of the RCTs gave details about randomisation procedures sufficient to enable assessment, and this did not improve over the 15 year period. The failure to give adequate information about randomisation procedures was more noticeable among single centre (125/204 (61%)) compared to multicentre (17/45 (38%)) RCTs. Table 6 shows the assessment of quality criteria in each of three successive five year periods. Of studies that gave sufficient information for assessment, an increasing number were scored in the highest category for attention to proper randomisation procedures, but a decreasing number scored in the highest category for completeness of data in the primary analysis over the 15 year period. A higher proportion of RCTs in respiratory medicine failed to give adequate information on randomisation 60/82 (73%) compared to trials in other areas 82/167 (49%) (table 7).
Assessment of key quality parameters of randomised controlled trials by year of publication
Assessment of key parameters of trial quality by speciality area
The consent rate was not reported in 112 trials (45%). In studies in which consent rates were reported, the rate was reported to be 100% in a very high proportion (111/137 (81%)) of studies. Consent rates varied with the study setting: of those that reported the rate, the consent rate was 100% in 51/57 (90%) inpatient studies, 51/65 (78%) outpatient studies, and 9/15 (60%) community studies. Consent also varied by specialty area with 100% consent rates reported (in studies that reported consent rates) by 23/34 (68%) general paediatric trials, 29/36 (81%) respiratory trials, 27/28 (96%) neonatal trials, and 32/39 (82%) in other speciality areas.
Forty five trials (18%) were multicentre RCTs, 13 published in 1982–86 (16% of all RCTs in this period), 13 (18%) in 1987–91, and 19 (20%) in 1992–96. Thirty (67%) of these trials were UK based compared to 57% of single centre RCTs. The proportion of RCTs that were multicentre in the various paediatric specialty areas was as follows: general paediatrics 23%, respiratory paediatrics 15%, neonatal paediatrics 11%, and other specialty areas 14%. The sources of funding for these RCTs were pharmaceutical companies in 23 (51%), research councils in 13 (29%), government sources in 5 (11%), charitable agencies in 7 (16%), and other specified source or source not given in 2 (4%). This pattern of funding was not significantly different from that for single centre RCTs. Where there was sufficient information to assess the quality of trial design according to the quality criteria noted above, the proportion of RCTs that scored the highest quality ratings in the following areas were: quality of random allocation, multicentre 46%, single centre 51%; blinding of outcome assessment, multicentre 44%, single centre 41%; and completeness of data in the primary analysis, multicentre 27%, single centre 45%. Table 8 shows the sources of funding support for the published trials by individual specialty areas.
Source of funding by speciality area (where details given)
Discussion
RCTs from a broad range of paediatric practice were published. RCTs in general paediatrics (69/249 (28%)) and respiratory paediatrics (82/249 (33%)) are the most frequently published in theArchives of Disease in Childhood. It is not clear to what extent this reflects the relative numbers and quality of RCTs submitted, or the editorial policy of the journal. Fifty nine per cent (148/249) of RCTs were carried out in the UK. The proportion of RCTs from outside the UK has increased over the 15 year period, with an increasing number of RCTs from other European countries and from Asia (accounting for 35% of all RCTs published in 1992–96 compared to 19% published in 1982–86).
Most RCTs reported in the Archives of Disease in Childhood recruited a very small number of children. Approximately half of all RCTs recruited fewer than 40 children in total (thus fewer than 20 to each treatment group). There are a number of identified benefits from small clinical trials,11nevertheless it is equally clear that these have inadequate power to detect small or moderate treatment effects and result in a significant chance of reporting false negative results.11 12 Only 18% of RCTs had multicentre collaboration. It may be that larger and more powerful multicentre trials tend to be reported in other paediatric journals, nevertheless it appears that there is a need to encourage multicentre collaboration to address important therapeutic questions in paediatric practice. Some national mechanism within the UK should be considered to foster this, perhaps under the auspices of the new Royal College of Paediatrics and Child Health. There were very few RCTs with a factorial design. This is likely to be caused in part by the small sample sizes in the reported RCTs but may also reflect an unfamiliarity with this efficient RCT design, which can evaluate more than one intervention and interactions among interventions within a single trial design.
There were relatively few RCTs (31; 12% of all RCTs) published that were done in the community setting. The reasons for this are not clear. Community paediatricians may submit RCTs to more specialist journals, they may have difficulty raising funding for RCTs, or they may be less active in research into treatment interventions in paediatric practice than are other paediatricians in more well established paediatric specialties. The great majority of published RCTs that took place in the community setting were in the area of general paediatrics. This may reflect the greater ease of recruiting patients in other areas of paediatric practice from outpatient specialty clinics or inpatient settings.
Only 84/176 (48%) RCTs involving pharmaceutical interventions were placebo controlled. More than half of these were in respiratory paediatrics and 47/84 (56%) were funded by pharmaceutical companies. This may reflect the fact that it is difficult to obtain adequate placebos without the collaboration of pharmaceutical companies. Publication of details of other sources of assistance in preparing suitable placebos might be helpful.
There is a need for better reporting of fundamental details of RCT design. The proportion of RCTs reporting details of key quality parameters has not increased over the 15 year period. This is disappointing given the recent widespread dissemination of information on RCT design and quality. This information is necessary to allow readers to interpret RCT findings and to assess their relevance to their own clinical practice. There are accepted quality criteria published for the key parameters detailed in this review: randomisation procedures, control of blinding of outcome assessment, and completeness of data entered into the primary analysis.6 7 TheArchives of Disease in Childhood should consider requiring these details to be given as part of its editorial review procedures [see Introduction].
We believe that consent rates should be explicitly stated (currently about half of all reports do not state this). We are concerned that consent rates appear to be extremely high in the great majority of RCTs. The fact that 90% of all RCTs in inpatient settings and 96% of all RCTs in neonatal paediatrics reported 100% consent may raise the question as to whether such consent is truly informed. It seems likely that if parents were fully informed of their options and freely able to choose not to take part at least some would choose not to, for a variety of reasons. It is possible that investigators who published RCTs with 100% consent rates were more likely to include a note of the consent rate in the publication, believing this to be an indicator of the quality of the RCT. An alternative, although perhaps less likely, explanation of these findings is that practices varied among settings with respect to application of inclusion/exclusion criteria. It is possible that failure to give consent was considered as an exclusion criterion (rather than being recorded as eligible for study but refusing to give consent), particularly in RCTs carried out in inpatient settings and in RCTs involving neonates. We recommend that this should not be done but that consent rate should be reported separately and, before acceptance of the article, authors should be invited to comment on the consent rate if it is reported to be 100%.
Thirty per cent of RCTs reported negative results. This is somewhat reassuring as reviews of RCTs in some other journals have reported very high proportions of RCTs with positive results suggesting publication bias. This can be caused by factors related to the journal in question or the authors, and has been discussed recently.13
The capture of RCTs by medline search strategies is improving, principally because of improved indexing of articles with respect to RCTs.3 8 The sensitivity ofmedline searching for RCTs published inArchives of Disease in Childhood was considerably higher than values reported with other journals in 1994.8 However, this is still at best only about 90% of RCTs, and in some paediatric specialty areas considerably less than this; therefore, there is still scope for improved indexing of original articles published in Archives of Diseases in Childhood so that RCTs are more completely identified.
Addendum
Citation information from this handsearching study have been forwarded to the Cochrane centre in Baltimore, details of the RCTs can be obtained from them. Anyone interested in contributing to further handsearching of Archives of Disease in Childhoodwill be warmly welcomed and should contact Dr H Campbell.
Acknowledgments
We thank Dr K Cheng of the Cochrane Cystic Fibrosis Group, Alder Hey Children’s Hospital, Liverpool for her assistance and advice in the handsearching exercise, and Professor M Preece, Institute of Child Health, London for forwarding to us a note of all RCTs published in the Archives of Disease in Childhood over the period 1984–92 which he identified by handsearching.
References
Editors’ comments
The Editors are always pleased to receive submissions that report a randomised controlled trial (RCT).Archives of Disease in Childhood, like many scientific journals, subscribes to the CONSORT statement—the consolidated standards of reporting trials—which was published by theJournal of the American Medical Association(1996;276:637–9).
Our referees will be asked to apply the CONSORT table and flow diagram when assessing RCTs. Authors are advised to follow these criteria as closely as they can. Further details are available onhttp://www.ama-assn.org.
Progress through the various stages of a trial, including flow of participants, withdrawals, and timing of primary and secondary outcome measures. “R” indicates randomisation.