Reliable

If something is described as ‘reliable’, what springs to mind? Dull but dependable? Solid and sure? It is a word that gets bandied about with respect to clinical studies, and I have often wondered quite what the user of the word means, apart from some kind of overarching sense of 'good science stuff'.

When I think of a study and its findings being reliable, I tend to use this as a conflation of a series of ideas: (1) that the study describes truthfully what was done in the world; (2) that the results are likely to be reproducible—if someone did the study again, they would get similar results because everything is explained well enough to understand it; (3) that the results of the study are (clinically) precise; the mathematical uncertainty in how effective the treatment, test or prognostic feature is does not swing me clinically from a ‘use’ to a ‘don't use’ position; and (4) the outcomes are valid; reflections of something in the world which is meaningful to a patient, such as quality of life, or a proxy for it, such as shrinkage on a radiologically measured tumour.

As with many compound ideas, the concept can be useful to shortcut explaining different elements of ‘good’. This comes with a price though, as describing something as ‘not particularly reliable’ does not give insight as to what aspect of ‘not good’ sits within the report. We need more detail on which aspects do not seem to be good enough to judge how best to interpret and use a study in practice.