Statistics from Altmetric.com
With interest I read the paper by Powell et al1 on the development of a questionnaire to describe respiratory symptoms in infants and preschool children. Because of the age of the children this is a difficult topic, and the authors are to be congratulated for their attempt. However, a number of questions arose when reading their paper.
First of all, the method for assessing test-retest reliability is questionable. The method, originally developed in the field of psychology, was used to see whether, when measuring some personality trait, assessing it repeatedly would give the same results. The kappa values for agreement in this area are usually in the range 0.70–0.90. Respiratory symptoms are not personality characteristics, and cannot be assumed to be stable. So when assessing symptoms over the previous three months, with two weeks interval, a change may be due to what statisticians call “measurement error”, but also to a change in symptoms. A related issue is the interpretation of the results. In the paper, the authors mention one kappa score below 0.40, but they fail to mention that the majority of other items were below 0.60 (reliability results from tables 1 and 2, accessible from ADC Online). In the abstract the authors conclude that the short term reliability is good, but this certainly overstates the results. It is not clear why the authors have chosen to compare the 20 referred children in whom a diagnosis of asthma was made to the 42 children from the newborn cohort. Why not compare them to the referred children who were not labelled as having asthma? Finally, it is unclear why in table 1 (paper version) the reader cannot reproduce the figures in the last column from the previous columns.