Article Text

PDF

A comparison of findings on parents’ and teachers’ questionnaires, and detailed ophthalmic and psychological assessments
  1. A R O’Connor1,
  2. T J Stephenson1,
  3. A Johnson2,
  4. S D Wright1,
  5. M J Tobin3,
  6. S Ratib4,
  7. A R Fielder5
  1. 1Division of Child Health, University of Nottingham, UK
  2. 2National Perinatal Epidemiology Unit, University of Oxford, UK
  3. 3Department of Special Education, University of Birmingham, UK
  4. 4Trent Institute for Health Services Research, University of Nottingham, UK
  5. 5Department of Ophthalmology, Imperial College School of Medicine, London, UK
  1. Correspondence to:
    Dr A R O’Connor
    Division of Orthoptics, Dept of Allied Health Professions, Thompson Yates Building, Quadrangle, Brownlow Hill, Liverpool L69 3GB, UK; annaocliverpool.ac.uk

Abstract

Background: Questionnaires are important tools used to gain information about health and level of function in different domains.

Aims and Methods: To determine the degree of agreement between questionnaires, administered to parents and teachers, and ophthalmic and psychological examinations in a cohort of 309 low birth weight children (<1701 g) at age 10–13 years.

Results: A total of 90.9% of cases showed agreement between the question on distance vision and clinical assessment, and agreement for the near vision question was 83%. However, the correlation on an individual basis was only fair (kappa = 0.46, distance vision) to poor (kappa = 0.2, near vision). The overall agreement for the questions on cognitive ability was better than the correlation, whereas the questions on reading and mathematical ability showed low agreement and low correlation.

Conclusion: Questionnaire assessment of vision and cognitive ability is more suitable for studying the outcome of a large population than for identifying deficits in individuals.

  • psychological
  • questionnaires
  • vision
  • low birth weight

Statistics from Altmetric.com

Babies born very preterm are at risk, in later childhood, of experiencing morbidities in several spheres. This information is needed by parents, and by those providing health, social, and educational services. Neonatologists also require this information to understand better the factors influencing the outcome of preterm birth. Undertaking the comprehensive multidisciplinary review necessary to assess and quantify protean morbidities is complex, expensive, and time consuming when carried out in person by professionals such as paediatricians, psychologists, and other specialists such as ophthalmologists or audiologists.

There has been considerable interest in finding alternative simple ways of following up large numbers of children by using parent completed questionnaires as a source of information about their child’s health and level of function in different domains.1–6 Parent completed questionnaires on the behaviour of their child are frequently used in both clinical and research settings,7 and as children achieve school age, the views of teachers of a child’s academic ability and behaviour can also be sought.3,8

The type of information obtained from each source will differ. Parents know their children well and are a potentially rich source of information. This may more closely reflect the child’s typical behaviour and ability than a single assessment of a child by a stranger, using tests that may not reflect ability to function in everyday life. On the other hand, parents and teachers may not be able to provide diagnostic information or to describe the characteristics of more subtle impairments, whereas a detailed assessment will not only provide these but also quantify the level of reduced function within an area.

A large follow up study of a cohort of 10–13 year old children born weighing less than 1701 g was mounted to explore the relation of ophthalmic function and cognitive ability and school performance.9,10 This study included a detailed eye examination and assessment by a psychologist and presented an opportunity to compare the findings on these assessments with the responses on a questionnaire completed by parents and a second questionnaire completed by the child’s teacher.

METHODS

Between 1 July 1985 and 31 May 1987, a prospective study of the ophthalmic outcome of low birth weight infants was undertaken in the five neonatal units serving the areas of Leicestershire, Nottinghamshire, and Southern Derbyshire Health Authorities. All infants who survived three weeks, with birth weights of less than 1701 g, plus any larger siblings of multiple births where at least one infant reached the criteria, and were admitted to one of these five neonatal units, were enrolled in the study (n = 610).

A follow up study of the long term ophthalmic outcome was carried out at age 10–12 years; 309 of the original cohort were assessed. Ophthalmic tests used fell into three broad categories. First, the following tests of visual functions: visual acuity charts for distance and near vision (logMAR charts); contrast sensitivity (Pelli Robson chart), stereoacuity (TNO plates); perimetry (Damato campimeter); and colour vision (desaturated D15 test). Second, tests of strabismus (cover test and prism tests). Third, tests of refractive state by cycloplegic refraction using the Retinomax K-plus instrument following cycloplegia with cyclopentolate 1%. Full details of the ophthalmic examination results are presented elsewhere.10 To summarise the ophthalmic data for the purposes of this study, ophthalmic morbidity was defined as visual acuity (with spectacle correction) below the accepted norm (0.0 logMAR), or the presence of strabismus, myopia, or visual field defect.

One year after the ophthalmic assessment further funding was awarded for a psychological assessment, so the parents were approached again to gain consent for this additional examination. Four children who had an ophthalmic assessment were severely disabled; they were not contacted as they would have not been able to perform the tasks on the psychological assessment.

Two hundred and forty of the 309 children who had an ophthalmic examination consented to an assessment by a psychologist. The psychological assessment included tests of cognitive ability (selected scales from the British Ability Scales—BASII)11 and reading ability (Neale Analysis of Reading).12 The overall measure of the BASII is the General Conceptual Ability (GCA) score. The GCA score is comprised of the summed scores of three clusters: verbal ability (composed of the subscales for Word Definitions and Verbal Similarities); non-verbal reasoning (composed of the subscales for Matrices and Quantitative Reasoning); and spatial ability (composed of the subscales for Recall of Designs and Pattern Construction). The Neale Analysis of Reading Ability–Revised is an oral reading test to assess ability of children aged 6–12 years to read aloud in three domains: reading accuracy, reading rate, and reading comprehension. Full details of the psychological assessment results will be presented elsewhere.

Questionnaires were sent to the parents of the 309 children immediately following the ophthalmic assessment, or completed during the ophthalmic assessment. These included questions about the parent’s perception of the child’s near and far vision, questions on their child’s ability to learn and remember new information, and their ability to solve everyday problems (questions in table 1). The questions were derived from a health status/disability measure developed by Saigal at McMaster University. These were also used in the development of the widely used Multriattribute Health Status Measure (MAHS II system).13 With the parents’ permission, questionnaires were also sent to the child’s teachers; these included questions on educational provision, behaviour,7,8 and academic progress. Teachers were asked to compare the study child with an “average” child of the same age, and in the same school class, and then to rate the study child’s ability in a number of areas using a four point scale: 1, above average; 2, average; 3, some difficulty; and 4, marked difficulty. As all the children that were eligible for inclusion in this part of the study were in mainstream education, this should limit the possible bias from comparison to an “average” child.

Table 1

 Questions given to parents

We compared:

  • The ability to see objects at near and distance as described by parents with the findings on the ophthalmological assessment

  • The parents’ response to questions on learning, remembering, and problem solving with the overall GCA score of the BASII

  • The teacher’s estimate of reading ability with the result of the Neale analysis of reading

  • The teacher’s estimate of mathematical skills with the result of the BASII quantitative reasoning subscale.

Permission for this study was obtained from Nottingham University Hospital Ethics Committee.

Statistical methods

The level of correlation between the detailed ophthalmic and psychological assessments and information on questionnaires was calculated using the weighted kappa statistic. Values for kappa range from 0 (indicating only chance agreement) to 1 (perfect agreement). Intermediate values are generally rated as: ⩽0.20 = poor, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = good, and 0.81–1.00 = very good. For comparison between the ophthalmic data and the questionnaire, each child, according to their corrected visual acuity and refraction, was categorised as: (1) normal binocular acuity requiring no spectacles, (2) requires spectacles, (3) moderately reduced corrected acuity, or (4) blind. The levels of acuity and refractive errors were chosen to correspond with the responses on the questionnaire. For distance vision the level for normal binocular acuity was ⩽0.3 logMAR and no significant myopia (no more than –0.5 dioptres), and for near normal acuity was <1.3 logMAR without glasses. For level 2 the acuity requirements were the same, but for distance acuity any child with significant myopia (−0.5 dioptres in both eyes) or high hypermetropia (>+5 dioptres in both eyes) was included in this group, and for near acuity any child who wore a hypermetropic correction or had myopia of 3 dioptres or more was included in this group. Level 3 was based solely on binocular corrected acuities and included any child with distance acuity worse than 0.3 logMAR or near acuity worse than 1.3 logMAR. Level 4 was for children with no recordable acuities.

We also calculated the sensitivity and specificity of the questionnaire responses in detecting children with (a) visual problems and (b) low scores on the administered scales. For comparison with the parent questionnaires the BASII GCA and the BASII matrices score were divided into three categories; the fourth category was excluded on the parent questionnaire, as those children would not be able to complete the BAS and therefore not have a score. The BAS scores are standardised, so the categories were divided into: (1) scores better than the mean or up to one standard deviation below the mean; (2) between one and two standard deviations below the mean; and (3) more than two standard deviations below the mean. For analysis the quantitative reasoning subscores and Neale reading score were divided into four categories: (1) one standard deviation above the mean; (2) up to one standard deviation above or below the mean; (3) between one and two standard deviations below the mean; and (4) more than two standard deviations below the mean.

RESULTS

Comparison of vision status

Of the 309 parents of children with an ophthalmic examination who were sent questionnaires, 287 were returned. Of the 22 non-responders, 19 had normal distance vision, and 20 had normal near vision. Not all questions were completed in every questionnaire; in 11 the question on distance vision, and in 5 the question on near vision were not answered. No child was blind, so that category has been omitted from tables 2 and 3.

Table 2

 Comparison of ophthalmic measurement of far vision and response on parent questionnaire

Table 3

 Comparison of ophthalmic measurement of near vision and response on parent questionnaire

There was overall agreement between parents and the ophthalmic assessment for 251 (90.9%) of the 276 children with completed questionnaires (kappa = 0.46, table 2). Thirty eight of 309 children were considered by ophthalmic assessment to have defective distance vision; three were among non-responders, in three the question was not answered, and in only 16/32 (sensitivity = 50%) did the questionnaire correctly identify these children. Of the 271 with normal distance vision, the response to the question identified 235 (specificity = 86.7%). Of the 25 discordant responses, six varied by two categories.

There was overall agreement in the level of near vision deficit in 235 (83%) of the 282 completed questionnaires returned (kappa = 0.21, table 3). On ophthalmic assessment, 49 (16%) of the 309 children tested had defective near vision. Of these, questionnaires were not returned for two, and 37 were not identified by the questions on near vision. Sensitivity of questions in detecting children with significant problem with near vision was 20.4%. Of those with no near vision deficit, 226 of 260 were correctly identified (specificity = 86.9%). Of the 47 discordant responses, three varied by two categories.

Comparison of cognitive status

Questionnaires were sent to all 240 of the parents of children who had a psychological assessment. Of these, 14 were not returned. Using the predefined categories, there was overall agreement for 167 of the 215 (77.7%) children who had psychological testing and also had completed parent questionnaires (kappa statistic = 0.59, table 4). Of the 174 children who had an overall GCA of 85 or more, 144 (82.8%) were considered by parents to learn and remember as well as other children. Of the 25 children with GCA score more than 2SD below the mean (<70), 20 were identified by parents as having difficulty with learning and remembering (sensitivity of these questions to identify children with GCA <70 = 80%).

Table 4

 Comparison of parents’ response to question on learning and remembering and BASII GCA score (standardised mean GCA score = 100, standard deviation = 15)

The child’s cognitive ability was also explored using a question to parents on their child’s ability to solve problems. The matrices subscale of the BASII tests inductive reasoning, and tests the child’s ability to identify and apply rules. The overall level of agreement was 73% (kappa = 0.26). Of 164 children who scored 40 or more, 147 (89.6%) were considered to have no difficulty in this area (table 5). Of the 18 with a score >2SD below the mean (a score of <30), however, only seven were thought to have difficulty with problem solving by their parents (sensitivity = 39%).

Table 5

 Comparison of parent’s response to question on problem solving and BASII matrices subscore (inductive reasoning, identification, and application of rules) (standardised mean of the BAS subscore = 50, standard deviation = 10)

Comparison of ability at school

All the parents of the 240 children who had psychological assessments gave permission for the teacher to be approached, and 209 questionnaires were returned complete.

A number of children could not be scored on the Neale reading test as they had exceeded the upper age limit of the test. Of the 143 who were tested and scored for reading accuracy, teacher questionnaire responses were available for 130. The overall agreement was 49% (kappa = 0.51) (table 6). Of 106 children who scored 85 or more on the Neale reading accuracy test, 91 (85.8%) were rated as average or above average by the teacher. Twenty four scored below 85, and 19 (79.2%) of these were rated as having some or marked difficulty by the teacher.

Table 6

 Comparison of teacher assessment of reading accuracy and comprehension, and Neale reading accuracy and comprehension scores (standardised mean = 100, standard deviation = 15)

The Neale comprehension score was available for 136 children and teacher questionnaires responses for 125 of these. The overall level of agreement was only 33.6% (kappa = 0.36) (table 6). Of 85 children who scored 85 or above on the reading comprehension test, 71 (83.5%) were regarded as average or above average by the teacher. Forty scored below 85, and 21 (53%) of these were rated as having some or marked difficulty by the teacher.

A comparison of mathematical ability was available for 196 children. The overall level of agreement was 47.9% (kappa = 0.57) (table 7). One hundred and thirty scored above 40 on the BASII quantitative reasoning subscale. Of these, 91 (70%) were ranked as average or above average by teachers. All 18 children who scored below 30 were identified by teachers as below average.

Table 7

 Comparison of teacher assessment of mathematical ability versus BASII quantitative reasoning score (standardised mean = 50, standard deviation = 10)

DISCUSSION

Questionnaires to ascertain the health and abilities of infants and children are used in a number of ways. At an individual level, the interest is focused on the extent to which simple questions can identify specific impairments or levels of functional loss which are detectable by another method, usually by a professional using a recognised method of measurement. This is the basis of a screening tool, and in this setting questions need to be sensitive and specific in order to be useful to clinicians, therapists, and to the children and parents themselves.

When ascertaining the function of large groups of children, however, the focus is different. For example, in a large randomised controlled trial of an intervention offered to babies in the neonatal period, the interest is in detecting differences in outcome between the intervention and non-intervention groups. In this situation, convergent validity is less of an issue, providing the same level of imprecision occurs in both arms of the trial. So, for example, comparing the responses to questions to health visitors and a paediatrician’s assessment,4 showed considerable disagreement at individual level, but the differences in the frequency of the trial endpoints (severe disability) were remarkably similar.

The questions used in this study were based on those used in an earlier version of the Multiattribute Health Status Measure which has been well validated.14 It used closed questions, in which only a choice between provided answers is allowed. This approach is known to lead to discrepancies at an individual level15 as the limited choices exclude many possibilities. The questions to parents on visual status showed moderate agreement for distance vision and fair agreement for near vision, but the ability of these questions to identify children with low visual acuity for near and far vision was poor. Even among the seven children with binocular VA worse than 0.3 logMAR, the response in five was that the child could see at the normal level without glasses. A more detailed questionnaire in an adult population produced better results when compared to standardised tests.16 There are other widely used visual questionnaires including the National Eye Institute Visual Functions Questionnaire (VFQ).17 However, these questionnaires are all designed for use with adults and are not applicable for use with children as they relate to tasks that children do not perform.

There is a possible flaw of completing the questionnaires after the visual assessment, as there is a potential for influencing the parents by these results. However, as this assessment was at the age of 10–12 years, all of the children had had previous visual assessment, whether at the eye clinic or by the school nurse, so the parents already had information from health professionals prior to the visual assessment for this study. Therefore, even if all questionnaires had been completed without discussion of the visual assessment results, there is still a possibility for bias from previous eye tests which it is not possible to control for.

The sensitivities of the questions aimed at identifying children from within a cohort with cognitive or school difficulties ranged from 33% to 100% and specificities were all above 80%, apart from the question to teachers on mathematical ability which had a specificity of 70%. These estimates take into the account the impact of non-response and incomplete data on the performance of the questionnaire. The responses by parents tended to overestimate their child’s ability when compared with their performance on standardised tests of cognitive ability, whereas the responses on the teacher questionnaire tended to give a less optimistic view of their school performance compared with standard tests of reading and cognitive ability. These mismatches are perhaps not surprising as a single formal psychometric test may not reflect either the teacher’s or parent’s perception of how a child is performing at school. These views are formulated over prolonged periods and will also be influenced by other factors such as behaviour patterns and health problems. It is also possible that if teachers are aware of the increased risk of cognitive impairment among children who had a low birth weight, this may also give rise to observer bias.

Response bias is also a problem with all types of follow up, not least, with self completed questionnaires. Differential response by severity of disability has been shown in previous studies with a higher level of disability among non-responders than among responders.18 In this study there was a unique opportunity to compare the findings on the full ophthalmic and cognitive assessments among those who completed the questionnaire and those who did not. As is clear from data on non-responders given in the tables, in this study there was no evidence of a response bias by level of severity.

It is clear from this and previous studies that although very simple questionnaires can reliably detect severe disability,19,20 it is difficult to detect milder degrees of impairment and disability. This is unfortunate because these are impairments which can impact educational progress, and these children in whom the impairment remains unidentified may be denied the support they need. There is an interest in developing more sophisticated and detailed questionnaire methods—Ages and Stages,21 Vineland,2 and the children’s VFQ22—and further work is needed in developing and testing these in different age groups and different populations, but in the meantime ascertainment of outcome into childhood by clinical assessment remains a key part of evaluating fetal and neonatal interventions.

REFERENCES

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Archivist
    BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health