Original ArticleTesting a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity
Introduction
A systematic review can be defined as a “scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies” [1]. When undertaking systematic reviews, the risk of bias for the included studies should be formally assessed because this factor has a substantial impact on estimates of treatment effects and may affect the validity of systematic reviews [2].
The design of the studies that are included in systematic reviews plays a major role in determining the reliability and the validity of the estimates of treatment effects. The randomized controlled trial (RCT) is widely regarded as the design of choice for assessing the effectiveness of health care interventions [2]. However, some questions of interest cannot be answered by a review of randomized trials, and some interventions cannot be randomized or are extremely unlikely to be studied in randomized trials. For example, evidence of certain effects, such as long-term and rare outcomes, or outcomes that were not considered important when major randomized trials were conducted, cannot be adequately investigated in randomized trials. In these contexts, review authors may be justified in including nonrandomized studies (NRSs) [3].
The Cochrane Collaboration introduced a tool to assess the risk of bias for RCTs [3], and this tool has been widely used. This tool was not developed to address NRSs, and certain domains of this tool are not necessarily appropriate for NRSs. However, the general structure of the tool and its assessments appear to be useful templates to follow during the assessment of the risk of bias for NRSs [3].
Many instruments for assessing the risk of bias for NRSs have been created, and these instruments were systematically reviewed by Deeks et al. [4]. In their review, these authors started with 182 tools. After reducing this number to a shortlist of 14 tools, they identified 6 tools that possess potential utility for systematic reviews [5], [6], [7], [8], [9], [10], although none of the examined tools had been formally validated. The Methodological Index for Non-Randomized Studies (MINORS) is the only quality assessment tool for NRSs that has been validated [11]. However, MINORS has several limitations with respect to systematic reviews; in particular, this tool is scale based and may therefore be unfit for certain study designs, such as the before-and-after design.
The objectives of this study were to develop and validate a new risk-of-bias tool for assessing NRSs in systematic reviews.
Section snippets
Development
A team of three experts (S.Y.K., S.S.S., and S.K.H.) in the field of evidence-based medicine was formed to develop a new risk-of-bias tool for assessing NRSs in systematic reviews. The authors reviewed previous risk-of-bias tools for NRSs used in systematic reviews that were developed by Deeks et al. [4] and West et al. [12] and examined similar tools that have been used by various organizations, such as the National Institute for Health and Clinical Excellence (which uses this type of tool to
RoBANS
We developed RoBANS that uses a domain-based evaluation approach. RoBANS was developed to be used for the assessment of virtually all study designs except for RCTs. RoBANS contains six domains for the risk of bias (the selection of participants, confounding variables, the measurement of exposure, the blinding of outcome assessments, incomplete outcome data, and selective outcome reporting; Table 2). It is also applicable to the GRADE approach and can be used with primary NRSs to facilitate the
Discussion
The RoBANS that we developed is not a scale but is instead a domain-based evaluation tool that is compatible with the Cochrane risk-of-bias tool. RoBANS is also applicable to the GRADE approach.
RoBANS is a tool to evaluate the risk of bias in non-randomized studies during systematic reviews of the effects of interventions. Among NRSs, epidemiological studies, such as etiological or prognostic studies, are not significantly different from interventional studies. Thus, it will not be difficult to
Acknowledgments
This study was supported by a grant from the National Evidence-Based Healthcare Collaborating Agency and the Health Insurance Review & Assessment Service of the Republic of Korea.
References (31)
- et al.
Data collection instrument and procedure for systematic reviews in the “Guide to community preventive services”
Am J Prev Med
(2000) - et al.
Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration
PLoS Med
(2008) - et al.
Impact of study quality on outcome in placebo-controlled trials of homeopathy
J Clin Epidemiol
(1999) - et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
(1998) - et al.
Finding what works in health care: standards for systematic reviews
(2011) CRD's guidance for undertaking reviews in health care
(2009)- et al.
Evaluating non-randomised intervention studies
Health Technol Assess
(2003) - et al.
The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions
J Epidemiol Community Health
(1998) - et al.
Aid to the evaluation of therapeutic studies
Pediatrics
(1989)
Prostheses for primary total hip replacement: a critical appraisal of the literature
Int J Technol Assess Health Care
Quality assessment tool for quantitative studies: effective public health practice project
Data extraction for nonrandomized systematic reviews
Methodological index for non-randomized studies (MINORS): development and validation of a new instrument
ANZ J Surg
Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47
Cited by (0)
Conflicts of interests: The authors declare that they have no conflicts of interest with respect to this study.
Authors' contributions: S.Y.K. designed and conducted the study and wrote the manuscript. S.Y.K., S.S.S., and S.K.H. developed the new tool and its instructions. H.J.S., Y.J.L., and J.E.P. participated in validating the new tool. B.H.J. and H.J.S. assisted in the performance of the study. All of the authors contributed to the conception and design of the study and read and approved the final manuscript.