New patient-reported experience measure for children with allergic disease: development, validation and results from integrated care

Objectives To develop and validate a new allergy-specific patient-reported experience measure (PREM) for children and their parents, and to collect feedback in an integrated care setting. Design Two allergy-specific PREMs were produced using focus groups, cognitive testing, two prospective validation studies (collaboration: Royal College of Paediatrics and Child Health, Picker Institute Europe, Imperial College/London): ‘Your Allergy Care’, for children aged 8–16 years; ‘Your Child's Allergy Care’, for parents of children aged 0–7 years. Setting Community event, primary/secondary/tertiary allergy care settings. Main outcome measures Performance of PREMs in validation study; reported experience of allergy care. Participants 687 children with allergic conditions and their parents/carers. Results In total, 687 questionnaires were completed; 503/687 (253 child; 250 parent) for the final survey. In both surveys, demographic variations were not associated with differences in results. Although 71% of patients reported one or more allergic conditions (food allergy/eczema/hay fever/asthma), 62% required multiple visits before receiving final diagnosis. Overall, patient experience was good for communication with patient/parent, competence and confidence in ability, and 73% felt looked after ‘very well’ and 23% ‘quite well’. Areas for improvement included communication with nurseries/schools, more information on side effects, allergic conditions and allergen/irritant avoidance. Allergy care in primary/emergency care settings was associated with higher problem-scores (worse experience) than in specialist clinics. Conclusions These new PREMs will allow allergy-specific patient experience reporting for children and parents and help identification of priority areas for improvement and commissioning of care. Efforts towards better allergy care provision must be targeted at primary and emergency care settings and underpinned by improving communication between healthcare providers and the community.


INTRODUCTION
Allergic conditions affect up to a third of children in the UK at some point during childhood making them the most common chronic childhood ailment. 1 Many children suffer from multiple allergic comorbidities, such as eczema, asthma, food allergies and hay fever. Access to specialist allergy care is difficult due to underprovision of services, as outlined in successive national reports. [2][3][4][5] Once accessed, there is evidence of considerable variability in the quality of allergy care, which is likely to negatively impact on clinical outcomes, patients' experiences and safety. 6 7 In response, the Department of Health commissioned the Royal College of Paediatrics and Child Health (RCPCH) to define care pathways for children with allergic conditions, including the development of allergy-specific patient-reported experience measures (PREMs). 8 PREMs allow patients to report on their experience of care rather than health outcomes ( patientreported outcome measures (PROMs), eg, quality of life tools). 9 Clinicians, service providers and commissioners can and should use PREMs to collect feedback as a key tool for evaluating current care and for shaping future service provision. 10 11 What is already known on this topic ▸ Patient-reported experience measures (PREMs) are important for evaluating care but very few condition-specific PREMs are available and even fewer for children. ▸ Children are able and willing to report their own experience of care, and it often differs from that of their parents. ▸ Allergy care for children is difficult to access and provision is patchy despite it being the most common chronic childhood condition in the UK.

What this study adds
▸ This study presents the first disease-specific PREM for children with allergic conditions, an essential tool and key quality indicator for future planning and commissioning of services. ▸ One-third of children with allergic disease have to visit four to eight healthcare professionals before receiving a diagnosis and complete treatment, suggesting significant training needs. ▸ Information needs are well described but frequently not met, offering an immediate area for improvement of children's allergy care.
Although there are young-patient feedback tools that are generic to services, [12][13][14][15] few are condition-specific, and the aim was to create a tool developed by children for children and by families for families to measure allergy-related patient and parent experiences of care.

METHODS
The PREMs were developed collaboratively by the RCPCH Allergy Care Pathway Project Team (for the stakeholders' list, see ref. 8) and the Picker Institute Europe using established survey-development methodology (figure 1). 16 17 In this report, Figure 1 Flow chart of patient-reported experience measure (PREM) development. 16  the term 'allergy care' encompasses any care for allergic conditions, delivered in primary, secondary and/or tertiary care, as well as emergency care settings, and the term 'parent' is used for parent/carer.

Sequential steps of PREM development, validation and testing Focus groups
Allergy care service users (children (8-11 years), young adults (15-21 years), parents of allergic children (0-11 years); see online supplement 1) were invited to give their views on their allergy care; this was aided by an experienced facilitator and interview topic guide. Main themes were identified and formed the basis of question development. Two PREM surveys were developed: one for completion by parents of children aged 0-7 years and the second for completion by children aged ≥8 years, the latter with an additional shorter parents' section.

Cognitive testing
This iterative process tests understanding of each version of the survey among the target population (ie, young allergy patients/ their parents) and ensures they can respond accurately. Cognitive interviewing refines a survey tool and is repeated until the wording, comprehension and volume of questions are fit for purpose.

Pilot study (version 1)
The initial pilot survey was in-field for 8 weeks in a tertiary allergy centre. Trained staff handed out the surveys (hand-out methodology 18 19 ) and were available to support participants during completion.

Validation study (final version)
The refined PREMs, 'Your Allergy Care' (child survey: 22 child questions; 13 parent questions; free text comment box) and 'Your Child's Allergy Care' ( parent survey: 34 questions; free text comment box), were handed out at a community event in primary/secondary/tertiary care clinics.

Analysis (each version)
The survey data were analysed using Picker 'problem-scores'. These show the percentage of patients for each question whose response indicates that a particular aspect of their care could have been improved. This summary measure can assist with prioritising areas for improvement. Problem-scores are calculated by combining the non-optimum response categories, for example, if a question is answered with 'Yes', 60%; 'Yes, sometimes', 25%; and 'No', 15% the problem-score will be 25% +15%=40%. Large scores highlight potential problems, and low scores reflect better performance.

Statistical analysis
The sample size (250 completed of each survey) is based on evaluation of tetrachoric correlations, which are appropriate for problem-scores. 20 The required sample size to detect a correlation of at least 0.3 is 250 (α=0.05, 1-β=0.8). Due to the selfcompletion method, missing responses are inevitable and percentage calculations exclude respondents that did not answer. Item frequencies were examined for the proportion of missing or non-evaluative responses, to detect early drop-out from the questionnaire and to identify potential ceiling effects. Item variances were examined to ensure questions provided sufficient score differentiation. Interitem tetrachoric correlations were computed to examine the relationship between questions and the overall structure of the questionnaire. Correlations were computed between each evaluative question and the 'overall rating' question. Score variation was investigated using general linear models with gender, ethnicity and who completed the questionnaire (child or parent) as main effects. Wherever appropriate, level of association (Kendall's τ-b) was calculated between corresponding parent and child responses.
The project falls under service evaluation, approved by the relevant institutional review board (Imperial College Healthcare NHS Trust)-it was not 'research', that is, not exploring specific hypotheses but instead asking people to provide feedback on their care. Therefore, ethical approval was not sought. The project work conformed to the Market Research Society's Code of Practice, for example, informed consent was obtained from all cognitive interview and focus group participants prior to their involvement. These were cognitively tested with five parents and five children (aged 2-14 years), amended accordingly and then completed by 74 children (and parents) and 110 parents attending a tertiary allergy clinic. Interim validity-testing performed on this sample, despite small numbers, revealed the following for the child (n=74) and parent (n=110) V1 surveys: ▸ a large number of questions had been missed out and ▸ correlations showed a wide range of relationships between items: some highly and some negatively correlated. Key findings from this first pilot V1 survey include: Children mostly approached their general practitioners (GP, 51.5%) or emergency departments (24.3%) for their first allergy care. While 38.5% of parents felt that their GP did not know enough about their child's allergic condition, 71.3% of parents and 85.7% of children rated their/their child's overall allergy care (any care setting) as good/excellent, and 6% of the parents and 1.4% of children as poor/fair. Communication between healthcare professionals (HCP) was rated not very good/not at all good by 20.2%; this increased to 28.4% for communication between HCP and nursery/school. Moreover, 61.5% (64/110) of parents reported that their child had an action plan but only 3 had it drawn up in primary care.

PREM development, testing and refinement
The surveys were refined based on interim analysis results and with input from stakeholders (questions trimmed, clarified, duplicates removed; number of questions brought down from 89 to 55, and finally 35). Further cognitive testing (seven parents; five children) was performed on the final version, resulting in: ▸ Child survey: 22 child questions, 13 parent questions; free text comment box ▸ Parent survey: 34 questions, free text comment box.
The final surveys are included as online supplement 3.

Final PREM validation and testing in various community and healthcare settings Final survey: demographics, allergic conditions and validation results
The final surveys include four domains mapping to focus-group themes (table 1) with the headings: 'Diagnosis', 'Seeing HCPs', 'Managing your/your child's condition' and 'Lifestyle'. Furthermore, they cover information about 'Your/your child's allergy' and 'Just a bit about you' (demographic information). A total of 503 surveys were completed ( parent survey, n=250; child survey, n=253; parent section incomplete for 13/253 (5%)). Table 2 shows demographic and clinical background information of the survey respondents. The majority of patients had long-standing chronic disease and a third of respondents had to visit HCPs four or more times to obtain a diagnosis.
Demographic variations were not associated with differences in results in either survey, non-response rate was low (<5%), there was no sign of early drop-out and questions showed good discrimination overall. Questions relating to 'how well looked after by most recent HCP' and ratings of overall care had low problemscores, indicating a ceiling effect in this format, that is, they were not good at discriminating 'good' and 'poor' experiences as they were answered mostly positively. There was moderate agreement (Kendall's τ=0.39-0.43) between specific items in the child survey and child survey parents' section (see online supplement 2).

What do our patients and their parents/carers tell us?
The results from parent and child surveys were moderately similar for most items. Figure 2A, B summarises results and problems-scores. Items where parent and child responses varied notably are presented separately within figure 2A, B. Overall, patients/parents felt that they and their conditions were well looked after by HCPs. Areas for improvement (high problem-scores) were:

Figure 2B
Continued ▸ communication (i) among HCPs and (ii) between HCPs and nursery/school; ▸ information giving, in particular on conditions themselves, allergen/irritant avoidance and medication side effects; and ▸ access to support, such as a phone number to contact HCPs and/or information on support groups. Parents expressed more interest in support groups and information on emergency treatments in the acute care setting than children ( figure 2A, B).
Participants who attended tertiary allergy clinics reported better experience than those who attended other healthcare settings for their allergy care (figure 2C). In comparison, problemscores were much higher in primary and emergency care settings.
Participants' free text comments from open-ended questions were converted into word clouds providing visualisation of the words used and give further insight into current experiences of allergy care ( figure 3A, B).

DISCUSSION
Patients' experiences, along with outcomes, are at the heart of healthcare provision and there is increasing national emphasis on PREMs being used by service users, service providers and commissioners as key quality indicators. 11 21 However, while there are many PROM tools for children, 22 23 there are few validated PREMs, and these tend to assess children's experiences of a specific hospital visit (eg, the Picker Institute's acute paediatric surveys).
We present the first disease-specific, PREM for children with allergic conditions and report on experiences that children and their parents/carers have had when seeking and receiving allergy care in a variety of community and NHS healthcare settings.

The validation study was carried out in multiple settings
where children receive allergy care. Due to staffing constraints, the majority of PREMs were completed in a tertiary care centre (74%). It is likely that high satisfaction with care and specific HCPs is related to this, as families benefitted from an expert multidisciplinary team with generous time allocation. Comparison between tertiary and primary/community care confirms this, showing high problem-scores across domains for non-tertiary healthcare provision. 2. The number of surveys handed out was not recorded (multiple staff; multiple settings), which is a recognised drawback of hand-out methodology. Surveys were handed out and participants were encouraged to complete them 'there and then'. The estimated return rate was >70%, based on copies produced and wastage. Hand-out methodology or point-of-care administration is associated with higher satisfaction scores compared with postal questionnaires, 18 19 24 making positive response bias possible. The free text responses, collated in word clouds, reveal more clearly where patients experienced difficulties.  Although more than two-thirds of children in North West London have English as their second language, they attend English-language schools and questionnaire completion did not significantly differ between ethnic groups. However, it is likely that families with poor English-language skills will have worse experiences in accessing allergy care and therefore will be under-represented in this study. 26 It is therefore important that PREMs (ideally in relevant languages) are used in community settings, especially in inner city areas with high deprivation indices, to elucidate children's experiences there.
Rationale for the PREM: does it achieve its objective?
This PREM was developed by children for children and by families for families. It performs well in a variety of community and healthcare settings. A total of 162 children completed the questionnaires by themselves with low drop-out or missing data, suggesting that the surveys are appropriately constructed and children can and will use them. While parent and child responses appeared mostly congruent, some differences were apparent and need to be taken into account. 27 What are the messages from the first children's allergy PREM study?
The overall messages are the same for children and parents/ carers. Taking the highest 25% of problem-scores (reflecting worst experience), key areas for improvement include: 1. Access to care, as many children have to visit numerous HCPs before getting a diagnosis 2. Communication involving HCPs: HCP to HCP across healthcare settings; HCP to nursery/school 3. Information about side effects, triggers/allergens and their condition overall 4. Support via patient support groups and directly from HCPs via a contact phone number Problems with accessing allergy care are well recognised 2 3 emphasising a lack of allergy services in the face of huge demand. 1 Although 90% of children with asthma and allergies should receive treatment in primary care, this is not happening owing partly to lack of education in allergy during undergraduate and postgraduate training. 28 A survey of GP trainers identified that few felt competent in allergy diagnosis and treatment, 6 explaining the lengthy journey families have until their child is diagnosed. The situation is compounded by the pressure on primary care providers to be gate-keepers to specialist (secondary or tertiary) referrals 29 and by the lack of specialist services, leaving few referral options. Our data confirm the knowledge gap, with higher problem-scores in primary/emergency care-settings compared with specialist clinics. Initiatives such as the RCPCH care pathways 8 30-34 and 'Itchy-Sneezy-Wheezy' integrated care project (http://www.itchysneezywheezy.co.uk) with community-based allergy clinics and educational initiatives to improve competencies in all care settings are a step in the right direction as a recent report on general paediatric community clinics confirms. 35 Our study confirms problems around communication and information giving/sharing. However, some of the apparent unmet information needs may relate to poor patient recall after healthcare consultations. 36 Information provided on causes, treatments and avoidance strategies in relation to different allergic conditions is frequently complex and HCPs must ensure that they offer opportunities to increase uptake of information via web-based resources, phone apps and so on as well as clinic letters, including treatment plans, written in appropriately adjusted language.
Communication with nurseries/schools requires permission from patients/parents. 37 Including standardised personalised management plans (eg, nationally agreed Allergy Action Plans for Children 38 ) with correspondence to patients/parents explaining that it is their responsibility to share this with nurseries/schools may help. School-based interventions can improve health outcomes, such as asthma, 39 and future integration of care between education authorities, schools and healthcare providers with key roles for school nurses is critically important.
Our findings on information needs are not unexpected. Previous qualitative studies involving patients/parents with allergies identified common themes, such as medication use, effects/ side effects, trigger factors, avoidance and causes. 40 41 Future training of HCPs delivering services to patients with allergies should include these themes, and better signposting to further information needs to occur.

CONCLUSION
This new PREM for children with allergic disease allows children and their parents/carers to feed back their experiences of allergy care. Feedback will allow targeted education and improvement to existing services. Service users as well as service providers must be encouraged to use PREMs to inform their own practice and to shape future allergy services when negotiating with commissioners.