Distinguishing opinion from evidence in guidelines

How does expert evidence differ from expert opinion?

Evidence in this context can be defined as facts (actual or asserted) intended for use in support of a conclusion.5 An opinion is a view or judgment formed about something, not necessarily based on facts.6 For example, a patient might say: “I had prostate cancer detected by prostate specific antigen (PSA) screening and I am alive 10 years later.” That is evidence. It is not the same as saying: “PSA screening saved my life.” That is an opinion. Similarly, a clinical expert might say: “I operated on 100 patients with prostate cancer and none of them died from prostate cancer.” That is evidence. It is not the same as saying: “Prostatectomy is effective.” That is an opinion. In both cases, the opinions might be based on that evidence, but the evidence is clearly not the same as the conclusion.

Using expert evidence to inform guidelines

An expert can be defined as “a person who is very knowledgeable about or skilful in a particular area.”7 In this context, this includes patients and patient representatives who may have expertise and insight through experiencing a condition or intervention, as well as health professionals with clinical experience and expertise. Experts are often needed to interpret evidence, and people sometimes consider expert opinion as evidence. For example, the Canadian Task Force on the Periodic Health Examination in 1979 included expert opinion as the lowest level of evidence.8 This categorisation can still be found in hierarchies of evidence.910 However, expert opinion is not the same as evidence.

When experts inform the development of a recommendation, it is important to clarify the evidence that supports their opinions.11 Attention should be focused on the evidence that is obtained from experts, not on their opinions (box 1).

When expert opinion is based on evidence that is not available to other members of the panel, the expert should be asked to present that evidence (box 1). Other panel members can then make judgments based on that evidence rather than on the expert’s views and judgments.

This approach can be used across guideline development approaches, including processes that use the RAND/UCLA appropriateness method (RAM) or a nominal group technique.13 Expert evidence can be solicited in addition to the description of case reports, and can be used to inform descriptions of the case scenarios in the RAM when research evidence is limited.

In addition to conflating opinion and evidence, using expert evidence to inform guidelines presents several challenges. We describe each problem and our proposed solutions below, following the conceptual framework in table 1.

Not distinguishing between expert opinion and expert evidence

The distinction between expert evidence and expert opinion is similar to the distinction between the results of a research study and the authors’ conclusions, which include interpretation of the results. If an expert offers an opinion (a conclusion) and does not clearly describe the basis for that opinion—that is, the supporting evidence—it is not possible to know what the evidence is or how trustworthy the opinion is.

Expert opinion is nearly always based on evidence. The evidence can come from randomised trials or other systematically collected observations or experiences, such as cohort studies, case-control studies, or qualitative research. It also can come from unsystematically collected observations or experiences. There are two reasons why research evidence is privileged in guideline development processes. Firstly, systematic methods reduce the risk of being misled by bias or the play of chance. Secondly, transparently reporting the methods that were used to collect and analyse observations or experiences enables others to appraise the quality or certainty of the evidence.14

Conflicts of interest

A conflict of interest exists when a person’s secondary interests interfere with or influence judgments regarding their primary interests.15 The primary interest for members of a guideline panel is making an appropriate recommendation. Secondary interests include financial interests, such as research funding, consulting income, or stock ownership. They also include intellectual interests, such as their research and publications, or professional and institutional loyalties.16

Experts on guideline panels often have financial or intellectual conflicts of interest. Some will have conducted research and published papers relevant to the topic of the guideline, and some have financial interests. Conflicts of interest do not necessarily lead to biased presentation or assessment of evidence but they are associated with bias, creating an important risk and potentially affecting the perceived credibility of the guideline.1517

Principles for disclosing interests and managing conflicts of interest in guidelines have been published elsewhere.18 The same principles should be applied to expert evidence, as well as to participation in panel discussions and decisions.

Inadequate appraisal of expert evidence

The risk of bias for evidence obtained from experts is large, primarily because systematic and transparent methods are rarely used to collect, analyse, and appraise this evidence. Expert evidence can be affected by recall bias, when experts remember only selected facts, as well as other cognitive biases.19 In addition, expert evidence is subject to all of the same risks of bias as research evidence, and the risk of being misled by the play of chance. When the methods used to collect and analyse that evidence are not transparent, it is difficult to appraise its trustworthiness (box 3).

Our solution for this problem is to use systematic and transparent methods to collect and appraise expert evidence. These methods should be clearly articulated and shared with guideline group members at the onset of the guideline process. In other words, we suggest managing expert evidence in much the same way as research evidence. This means having clearly formulated questions, explicit methods for obtaining the evidence, explicit methods for appraising the evidence, and making the evidence public.

The first step in deciding about the use of expert evidence is to identify those questions that cannot easily be answered using research evidence and that are important for deciding what to recommend. These questions can be related to any of the criteria in a GRADE evidence-to-decision framework, including whether a problem is a priority (how important it is in terms of the potential benefits or savings), potential desirable and undesirable health effects of interventions, how people value the main outcomes, the resource requirements (costs) of interventions, effects on equity, and the acceptability and feasibility of interventions.2122 For tests, it can also include questions about linking test results to management decisions.23 Expert evidence from patients may be particularly important for judgments about how people value the main outcomes, including how the intervention is experienced, its acceptability, and its feasibility.

In guidelines for the catastrophic antiphospholipid antibody syndrome (box 1), we appraised the evidence in the same way as for a case series and rated the certainty of the evidence as very low using the GRADE approach because of the risk of bias and imprecision. Because recommendations were required and individual clinicians were basing their interventions on much smaller numbers of observations and were more prone to recall bias, the guideline developers thought this approach was the best way to assimilate the evidence.

Other types of evidence can be collected in the same way, using structured forms to collect data in writing from experts, regardless of the type of evidence provided (see web appendix for an example), aggregating it, and using appropriate criteria to appraise it. Critically appraised expert evidence can then be presented to a guideline panel in the same way as research evidence, with a concise summary of the evidence in an evidence-to-decision framework, and a link to a more detailed report from the experts that is made public. It can then be discussed by the panel in the same way as research evidence. Box 4 sets out the advantages of our approach.

Moving forward with expert evidence

The additional time and resources required to formally collect and appraise expert evidence may be a disadvantage for some guideline developers. Our proposed approach could also result in undue emphasis on expert evidence, although this risk should be mitigated by appraising expert evidence in the same way as research evidence. In fact, we would expect our approach to result in less weight being given to expert evidence or, as is often the case, expert opinion. Furthermore, we are not suggesting using expert evidence as a substitute for research evidence—only when research evidence is unavailable or inadequate.

We have tested the use of standardised report forms to collect expert evidence in a few situations.12242526 Both qualitative and quantitative evaluation of this approach could inform decisions about when and how to implement it, as well as providing more detailed guidance and more examples to facilitate its use. However, the problems that we have identified are common and it is important to address them. We believe there is a compelling rationale for our approach and we encourage others to test it.