Original Article
Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity

https://doi.org/10.1016/j.jclinepi.2012.09.016Get rights and content

Abstract

Objectives

To develop and validate a new risk-of-bias tool for nonrandomized studies (NRSs).

Study Design and Setting

We developed the Risk of Bias Assessment Tool for Nonrandomized Studies (RoBANS). A validation process with 39 NRSs examined the reliability (interrater agreement), validity (the degree of correlation between the overall assessments of RoBANS and Methodological Index for Nonrandomized Studies [MINORS], obtained by plotting the overall risk of bias relative to effect size and funding source), face validity with eight experts, and completion time for the RoBANS approach.

Results

RoBANS contains six domains: the selection of participants, confounding variables, the measurement of exposure, the blinding of the outcome assessments, incomplete outcome data, and selective outcome reporting. The interrater agreement of the RoBANS tool except the measurement of exposure and selective outcome reporting domains ranged from fair to substantial. There was a moderate correlation between the overall risks of bias determined using RoBANS and MINORS. The observed differences in effect sizes and funding sources among the assessed studies were not correlated with the overall risk of bias in these studies. The mean time required to complete RoBANS was approximately 10 min. The external experts who were interviewed evaluated RoBANS as a “fair” assessment tool.

Conclusions

RoBANS shows moderate reliability, promising feasibility, and validity. The further refinement of this tool and larger validation studies are required.

Introduction

A systematic review can be defined as a “scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies” [1]. When undertaking systematic reviews, the risk of bias for the included studies should be formally assessed because this factor has a substantial impact on estimates of treatment effects and may affect the validity of systematic reviews [2].

The design of the studies that are included in systematic reviews plays a major role in determining the reliability and the validity of the estimates of treatment effects. The randomized controlled trial (RCT) is widely regarded as the design of choice for assessing the effectiveness of health care interventions [2]. However, some questions of interest cannot be answered by a review of randomized trials, and some interventions cannot be randomized or are extremely unlikely to be studied in randomized trials. For example, evidence of certain effects, such as long-term and rare outcomes, or outcomes that were not considered important when major randomized trials were conducted, cannot be adequately investigated in randomized trials. In these contexts, review authors may be justified in including nonrandomized studies (NRSs) [3].

The Cochrane Collaboration introduced a tool to assess the risk of bias for RCTs [3], and this tool has been widely used. This tool was not developed to address NRSs, and certain domains of this tool are not necessarily appropriate for NRSs. However, the general structure of the tool and its assessments appear to be useful templates to follow during the assessment of the risk of bias for NRSs [3].

Many instruments for assessing the risk of bias for NRSs have been created, and these instruments were systematically reviewed by Deeks et al. [4]. In their review, these authors started with 182 tools. After reducing this number to a shortlist of 14 tools, they identified 6 tools that possess potential utility for systematic reviews [5], [6], [7], [8], [9], [10], although none of the examined tools had been formally validated. The Methodological Index for Non-Randomized Studies (MINORS) is the only quality assessment tool for NRSs that has been validated [11]. However, MINORS has several limitations with respect to systematic reviews; in particular, this tool is scale based and may therefore be unfit for certain study designs, such as the before-and-after design.

The objectives of this study were to develop and validate a new risk-of-bias tool for assessing NRSs in systematic reviews.

Section snippets

Development

A team of three experts (S.Y.K., S.S.S., and S.K.H.) in the field of evidence-based medicine was formed to develop a new risk-of-bias tool for assessing NRSs in systematic reviews. The authors reviewed previous risk-of-bias tools for NRSs used in systematic reviews that were developed by Deeks et al. [4] and West et al. [12] and examined similar tools that have been used by various organizations, such as the National Institute for Health and Clinical Excellence (which uses this type of tool to

RoBANS

We developed RoBANS that uses a domain-based evaluation approach. RoBANS was developed to be used for the assessment of virtually all study designs except for RCTs. RoBANS contains six domains for the risk of bias (the selection of participants, confounding variables, the measurement of exposure, the blinding of outcome assessments, incomplete outcome data, and selective outcome reporting; Table 2). It is also applicable to the GRADE approach and can be used with primary NRSs to facilitate the

Discussion

The RoBANS that we developed is not a scale but is instead a domain-based evaluation tool that is compatible with the Cochrane risk-of-bias tool. RoBANS is also applicable to the GRADE approach.

RoBANS is a tool to evaluate the risk of bias in non-randomized studies during systematic reviews of the effects of interventions. Among NRSs, epidemiological studies, such as etiological or prognostic studies, are not significantly different from interventional studies. Thus, it will not be difficult to

Acknowledgments

This study was supported by a grant from the National Evidence-Based Healthcare Collaborating Agency and the Health Insurance Review & Assessment Service of the Republic of Korea.

References (31)

  • D.E. Cowley

    Prostheses for primary total hip replacement: a critical appraisal of the literature

    Int J Technol Assess Health Care

    (1995)
  • H. Thomas

    Quality assessment tool for quantitative studies: effective public health practice project

    (2003)
  • G. Wells et al.

    Data extraction for nonrandomized systematic reviews

    (2003)
  • K. Slim et al.

    Methodological index for non-randomized studies (MINORS): development and validation of a new instrument

    ANZ J Surg

    (2003)
  • S. West et al.

    Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47

    (2002)
  • Cited by (0)

    Conflicts of interests: The authors declare that they have no conflicts of interest with respect to this study.

    Authors' contributions: S.Y.K. designed and conducted the study and wrote the manuscript. S.Y.K., S.S.S., and S.K.H. developed the new tool and its instructions. H.J.S., Y.J.L., and J.E.P. participated in validating the new tool. B.H.J. and H.J.S. assisted in the performance of the study. All of the authors contributed to the conception and design of the study and read and approved the final manuscript.

    View full text