METHODS

Design of the questionnaire

Using the standard questionnaires already in use, mainly for older populations,^4–10 a questionnaire was designed. The following features common to these were used: wheezing, cough and breathlessness frequency, within four concept domains of daytime symptoms, night time symptoms, impact of the symptoms on the child, and impact on the family. The suggestions of six respiratory paediatricians were used to help finalise the content. It was felt that long term parental recall would be unreliable, therefore most of the questions refer to the child’s symptoms over the preceding three months. A template based on a previous questionnaire was employed where each question has a stem and five possible graded responses.¹⁴ All questions were phrased so as to avoid the use of technical terms, leading questions, composite questions, double negatives, presuming questions, prestige questions, imprecise questions, and use of words open to interpretation.¹⁵

Assessment of questionnaire characteristics

Practicality

This is a qualitative process that assesses the ease of administration and acceptability of the questionnaire. A number of colleagues and parents of children attending clinics were asked to read the questionnaire; it was then refined further, taking into account the views of these individuals.

Response rate

This is the number of individuals returning the questionnaire and was determined by administering it to parents of children who were recruited at birth from the maternity unit, and also parents of children under 3 years of age referred to the outpatient respiratory clinic with chronic respiratory symptoms (questionnaires were sent prior to the visit for the parents to complete before the consultation).

Reliability

The two aspects of reliability that were studied were:

Test–retest reliability

The test–retest reliability is the stability of response, which is short term reliability over time (usually 1–2 weeks). This is assessed using Cohen’s kappa, weighted kappa score, and average correct classification scores for analysis. Kappa scores >0.41 are considered to show moderate agreement; those >0.61, good agreement; and those >0.81, very good agreement.^16,17

The parents of the infants recruited from birth were sent a further questionnaire two weeks after the first one was returned. The parents of the children referred to the respiratory clinics were asked to complete the questionnaire again while attending clinic prior to their consultation. Having two questionnaires per child enabled us to examine the short term reliability of questions.

Internal consistency

The internal consistency is the extent to which the questions in each underlying concept domain interrelate with each other (using Cronbach’s alpha and factor analysis). An alpha score >0.7 is acceptable, >0.81 is good, and >0.91 is excellent.¹⁸

The internal consistency of each of the four domains was examined from the responses to all the initial questionnaires collected. Using factor analysis, alternative domain concepts were developed and further exploration of construct validity was completed.¹⁹

Validity

Face and content validity

The assessment of face validity and content validity (whether the questionnaire covers all relevant aspects of the topic unambiguously) was made informally by the respiratory paediatricians involved in the study (NJS and CVP).

Criterion validity

There is no “gold standard” objective test that can be used to categorise wheezy infants in order to compare the questionnaire against. There are many clinical phenotypes described in the literature but the different possible diagnoses are still to be clearly defined. Gold standards could be airway hyperresponsiveness (which have difficulty with sensitivity, specificity, and reliability in their own right^20–22), breath sound analysis,²³ or the consultant respiratory specialist opinion. We have used consultant opinion as a gold standard (although there is certain to be variability among them), to examine the initial qualities of the scales derived from the previous section of analysis. To determine whether the questionnaire could differentiate a child with no or minimal symptoms from a child with enough symptoms to be labelled as having asthma, scores for infants from the newborn cohort and those who had been seen by a respiratory physician and labelled as having asthma (RPBDA) were compared using the two sample t test.

Sample size

Previous questionnaire studies have calculated reliability, internal consistency, and aspects of validity using between 40 and 160 subjects.^{8–10,12,14,24,25} Factor analysis needs to have a minimum of 100 subjects for adequate assessment.¹⁹ We thus aimed to recruit at least 50 patients with symptoms from the outpatient clinic and at least 50 patients from a newborn cohort. For more detailed assessment of criterion validity, many more subjects would be required.

Ethics

The study had the approval of the local research ethics committee and the parents gave written consent prior to completion of the questionnaires.

RESULTS

Response rates and demographics

The initial response from the newborns recruited was 128/200 (64%). Repeat questionnaires were received from 72/128 (56%). All the 42 outpatients completed an initial and repeat questionnaire. Thus 114 pairs of questionnaires were analysed. Twenty of the outpatients had been diagnosed as having asthma. The mean age of the newborn cohort at the time of completion of the questionnaire was10 months and that for the outpatients was 13 months (range of the whole group was 6–35 months). There were 60 (53%) males.

Reliability

Short term reliability

Most questions have a kappa score of greater then 0.4, which suggests moderate agreement; only one question on “noisy breathing from the throat”, a rather ill defined symptom, had a kappa score of 0.39 (95%CI 0.35 to 0.5), indicating only fair agreement.

Internal consistency

Cronbach alpha scores for the four domains are as follows: daytime symptom score (19 components) 0.95, night time symptom score (five components) 0.80, impact of symptoms on family score (four components) 0.91, and impact of symptoms on child score (four components) 0.87.

For more detailed examination of construct validity, using factor analysis on the 32 questions involving a scaled response, six initial principle components were identified. Question 24, about snoring, was the sixth component (eigen value 1.2) accounting for 3.7% of the variance; questions 17, 18, and 19, about fast breathing and noises coming from the throat, was the fifth component (eigen value 1.33) and accounted for 4% of the variance. These questions were excluded prior to further factor analysis on the remaining 28 questions.

On examination of the questions four new scales are suggested with higher Cronbach alpha scores. Scale one refers mainly to cough and questions associated with impact of symptoms on the family and affect on feeding and waking in the child (cough and impact score). Scale two is mainly symptoms of a rattly chest and wheezing, possibly activity related (activity wheeze and rattle score). Scale three refers mainly to shortness of breath with symptoms reducing the child’s activity (short of breath score). Scale four, although the weakest correlations, suggests another wheezing scale associated with upper respiratory tract infections (URTI) and nocturnal wheezing with symptoms making the child tired (wheezing with URTI score).

Criterion validity

Comparison of the replies from the 20 subjects who had been diagnosed as asthmatic by a respiratory specialist (RPBDA) was made with those from the 72 subjects identified from the neonatal population with no, or minimal symptoms. The table shows that all eight of the scales in the questionnaire will significantly differentiate a child with a physician diagnosis of asthma from a well child. Sensitivity and specificity for cut off scores for each of the scales, for differentiation between RPBDA and well children can also be calculated. Further detailed examination of the questionnaire with respect to criterion validity will require larger numbers of infants with different diagnoses.

DISCUSSION

We have developed a practical, acceptable respiratory symptom questionnaire for use in infants and preschool children. This had satisfactory response rates when administered to a cohort of postnatally recruited subjects and was easy to use in an outpatient population. Face and content validity were considered acceptable.

Short term reliability

We have shown that most questions have moderate to good short term reliability. We have calculated both weighted kappa scores (measure of agreement which allows for the possibility of chance agreement, but is affected by the prevalence of the positive responses) and average correct classification (a measure of repeatability which is relative to the prevalence of responses). These measures should both be calculated, to give complete information on repeatability; when the prevalence of a symptom is low, the kappa score may be low and the average correct classification is considered to be a more accurate estimate of the reliability.¹⁸ Only one question (about noisy breathing from the throat) had a kappa score of <0.41. This probably should not be used and indeed has been identified by the factor analysis as being non-contributory to the scales we have developed. The kappa levels obtained are similar to those in other questionnaires designed for older children, questions on wheeze usually being more reliable than questions on cough.^24,26 As has also been noted previously questions concerning symptoms with colds and upper respiratory tract infections were less reliable than questions on symptoms without colds.²⁴ All questions in our questionnaire had an average correct classification score above 0.7.

Internal consistency

Four initial concept domains were designed and the internal consistencies for their scores were acceptable, with the “impact on family” and “daytime symptom” scores having excellent internal consistency. Essentially this means that each score has items, which are closely related and have good within-scale, between-item correlations. The level of internal consistency for these four domains is similar to that reported by Usherwood et al in their questionnaire designed for older children.¹⁴

Factor analysis and construct validity

To further refine these scores, principal component factor analysis was used. This is a powerful technique for exploring the item correlation during score validation.¹⁹ Varimax rotation was used to minimise the number of variables that have high loadings on each factor, thereby simplifying the overall structure.¹⁹ Initial analysis suggested that within the 32 items there were six possible areas of correlation. The question (number 24) on snoring was an outlier and appeared to have little correlation with the other factors. The questions about the noise from the throat (numbers 17 and 18) and fast breathing (number 19) were also outliers and were excluded from analysis. These exclusions resulted in four new domains.

The first new domain mainly concerns coughing symptoms (“cough and impact score”). It is interesting that most of the “impact on the family” questions closely correlated with these coughing questions. This suggests that parents may be more concerned about coughing as a symptom (particularly at night) than wheezing or shortness of breath. The second new domain focuses on wheeze associated with activity and having a rattly chest (“activity wheeze and rattle score”), and the third domain was characterised by shortness of breath (“shortness of breath score”). All three of these new domains had Cronbach alpha scores of 0.95 or greater which suggests excellent, within-scale, between-item correlations. The fourth new domain is characterised by wheeze associated with colds or upper respiratory tact infection (“wheeze with URTI score”). This had a lower Cronbach alpha score of 0.79, which is still acceptable. One could hypothesise that these four new domains represent four phenotypes of respiratory symptoms, in the infant and preschool child with wheeze and cough. The next stage is to use these four new scales in a larger cohort of symptomatic subjects to examine whether different diagnostic phenotypes show different scores within these domains. As far as our present study goes, the questionnaire has strong construct validity and internal consistency with the eight concept domains used.

Criterion validity

Criterion validity was the hardest quality to assess as there is no objective measure or “gold standard” to compare with. We have coarsely assessed it using respiratory physicians based diagnoses of asthma (RPBDA) in an outpatient cohort. The eight questionnaire scales can significantly distinguish children with RPBDA from mildly symptomatic and well infants. Subjects with RPBDA had significantly higher scores on the eight scales compared to those who had minimal or no symptoms (no diagnosis of asthma; see table 1). We used ROC curves to examine the performance of each for the eight scales. All the scores had acceptable sensitivity (between 88.9% and 96.7%) for identifying RPBDA but some had unacceptably low specificity (such as the wheeze with URTI; 31.3%).

We have only studied a small cohort for the initial exploration of the questionnaire characteristics with a few subjects labelled as having asthma. Other questionnaires have used similar or smaller numbers for validation exercises.^24,25 However we have been able to describe differences between those with asthma and those without. The major problem for any asthma and respiratory symptom questionnaire is not having a gold standard for examining criterion validity. Pragmatically we have used respiratory physician based diagnosis of asthma as the gold standard. Bronchial challenge tests could be used as the external measure for criterion validity. However, these tests have many problems with their own reliability and relation to clinical symptoms.^20–22 More detailed study of criterion validity against a different gold standard, together with the responsiveness of the questionnaire is required, using a larger population and including children with the different phenotypes of wheezy illness.

Other than using the Flesh score, we have not attempted to examine the readability of the questionnaire in detail. Neither have we looked at the factors which may affect responses such as cultural background, language spoken at home, age, or gender of the person completing the questionnaire, relationship to the child, and level of education. These are clearly extremely important and need to be explored further.

Conclusions

We have developed a respiratory symptom questionnaire, for use in the infant and preschool child. It examines the impact these symptoms have on the child and family as well as exploring different symptom patterns. We have shown it to be acceptable and easily completed with good response rates. Face and content validity are acceptable. Most questions have good test–retest reliability. Eight symptom scores have been developed with high internal consistency and construct validity. These scores may reflect different phenotypes of respiratory symptoms within the infant and preschool child population. Using a respiratory physician’s clinical diagnosis of asthma as a gold standard we have shown good criterion validity for the eight scores. The next challenge in the development of the questionnaire is to use it on larger populations, to further explore the possible phenotypes suggested in this initial assessment, and to assess the responsiveness of the questionnaire to change in symptoms and clinical state within individuals.