The limitations of the kappa statistic method to measure inter-observer variation of categorical assessments are shown by means of a hypothetical example. Another method for assessing the inter-observer variation of categorical variables, the proportion of agreement, is used for the same example, and reasons why it is preferable to the kappa statistic are given. Since this method allows measurement of the inherent difficulty of carrying out a particular assessment, it has wide applicability in the introduction of new technology. If a proportion of agreement study shows poor inter-observer agreement for a new method, the technology must either be improved or abandoned.