Research ethics committees (RECs) in the UK aim to make decisions on research proposals in line with published guidance.¹ The importance of systematic and consistent decision making by RECs has been repeatedly emphasised, most recently in the national standard operating procedures for RECs,² prompted partly by European Union legislation (Directive 2001/20/EC of the European Parliament and of the Council of 4 April 2001 on the approximation of the laws, regulations and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use 2001³). Much of the evidence on consistency in decision making by RECs in the UK is based on studies of a single proposal being submitted to multiple committees and is authored by the investigators themselves, usually in response to frustrating experiences.^4–6 Little evidence exists on the extent to which RECs agree about proposals. We compared decisions on the same proposals made by different RECs.

METHODS

The three RECs in the Leicestershire, Northamptonshire and Rutland Strategic Health Authority were included in the project, which was conducted between February 2004 and February 2005. Members of the committees gave their oral consent to participation in autumn 2003. Purposive sampling was used to select applications for inclusion in the project. Sampling, which was conducted by EA, an REC administrator, aimed to represent different types of studies and different types of applicants. The different types of studies were as follows:

• Intervention: a study that intervenes in the normal clinical care of a patient or health service user.

• Non-intervention: a study that does not include an intervention but seeks to measure outcomes or processes.

• Qualitative: a study that uses distinctive qualitative research methods.

The types of applicants were:

• Novices: applicants who had not previously applied to the REC.

• Experienced: applicants who had submitted at least one previous application to the REC.

Over a 12-month period, each application was reviewed by all three RECs and all of them prepared a decision letter. Each application was assigned a “lead” REC before any committee considered the application, and applicants received the decision letter of the lead REC only. The two non-lead RECs were sent this application as a “dummy”, despatched as one of the many applications for each meeting, and not labelled as “the dummy” (although it may sometimes have been possible for members to guess). Over the course of the project, each committee reviewed 12 dummy applications and 6 applications as the lead committee.

Three decision letters for each application were generated for analysis. Under governance arrangements for RECs¹ and guidance on standard operating procedures,² there were four formal decisions available to RECs:

• Favourable: the application is ethically acceptable.

• Provisional: amendments to the application or further information are required before a final decision can be made.

• Unfavourable: the application is ethically unacceptable.

• Outside remit: the application is deemed to fall outside the remit of governance arrangements for RECs.

Patterns of agreement in decisions were assessed descriptively. Agreement was further assessed using the κ statistic, which indicates the proportion of observed agreement that cannot be explained by chance. The one “favourable” decision was grouped as “provisional” for this analysis.

RESULTS

Of the 18 applications, 11 received consistent decisions from all three committees (table 1). A provisional decision was the most frequent decision for all the three RECs. REC3 had a higher incidence of unfavourable decisions (five, compared with two in REC1 and REC2), but was also the only REC to give a favourable decision. REC2 was the only one to give an “outside remit” decision.

Of the seven applications that received inconsistent decisions, six received consistent decisions from two committees. One application received different decisions from all three committees. Of the six applications where two committees agreed, four received consistent decisions from REC1 and REC2, one received consistent decisions from REC2 and REC3, and one received consistent decisions from REC1 and REC3. All the three committees agreed on seven novice and four expert applications, and all three did not agree on three novice and four expert applications. The committees agreed on five non-intervention, four qualitative and two intervention studies, but did not agree on one non-intervention, three qualitative and three intervention studies.

An analysis of agreement of decisions between RECs yielded an overall measure of agreement of κ = 0.286 (95% confidence interval (CI) −0.06 to 0.73), indicating a level of agreement that, although probably better than chance, may be described as “slight”.

DISCUSSION

Our study provides evidence on the consistency of outcomes of decision making by RECs. It was limited by its location in a single strategic health authority. Variations in the type of applications submitted over time constrained the extent to which it was possible to select equal proportions of different types of studies and applicants, although a reasonable balance was achieved. More importantly, the difficulties of conducting this type of research, which necessarily results in additional burdens on RECs, limited the size of the sample that could be obtained. The wide CI shows a considerable uncertainty about the point estimate, and polarised response categories such as those found in our dataset make the interpretation of κ statistics difficult. This inherently low-powered analysis nonetheless deals with an important question in a subject in which it is difficult to obtain data.

Although 11 of 18 applications considered by three RECs in our project received consistent decisions, seven—over a third of the sample—received inconsistent decisions. This evidence raises the bigger question of the significance of consistency of decision making for various arguments.⁷ Disparities in process and outcome are a source of frustration and delay for researchers,⁸ besides the possibility that, in a context where research governance is constructed as a regulatory and managerial enterprise,⁹ variations in outcome will be seen as evidence of problems in performance. Further research on reasons for variations in decision making, including further analysis of the content of decision letters in our own study, is needed. Such research may help further understand the kind of issues identified by RECs as being “ethical”, the way to resolve them, and the kind of ideas and principles that seem to inform opinions about ethical issues. This kind of research can afford an exploration of whether disparities are due, as previous authors have suggested, to inconsistencies in moral judgements (which may be unavoidable, acceptable or even desirable) or due to irrationality or carelessness, or operation of conflicting interests (which should be reduced or removed).¹⁰