Background

This paper describes the arrangements put in place to monitor the delivery of the Childsmile programme. It outlines how academic departments, service providers and NHS organisations have cooperated in using a range of data collection methodologies to establish a comprehensive monitoring process which continues to shape the direction of the programme. It then reviews the evidence regarding the growth in the numbers of participating children, families, practices, clinics, nurseries and schools, and in the delivery of toothbrushing and fluoride varnish treatments, over the 2006-2009 period.

The companion paper to this1 describes in detail the initial conception of Childsmile as a longitudinal prevention programme which seeks to break the link between service contact and acute episodes requiring restoration or extraction. This shift towards more preventive and anticipatory care implies the need to involve families of children from infancy onwards.2 In addition, the programme aims to reduce the marked and longstanding oral health inequalities due to deprivation.3

The Childsmile programme has three phases: 1: Demonstration Phase (2006-2008) - West: Childsmile Practice; East: Childsmile Nursery and Childsmile School; 2: Interim Phase (2009-11) - development of an integrated programme across Scotland; 3: Mainstream into dental services. At the end of the Demonstration Phase the programme was reconfigured into four interlocking and coordinated elements: Childsmile Practice, Nursery, School, and Core, all with specific monitoring arrangements. The second part of this paper presents data on each of these elements in turn. As the paper's title makes clear, these data represent process measures summarising the delivery of the programme: outcome measures relating to any improvements in child oral health and to the cost effectiveness of Childsmile will be covered in future papers.

From the start of the Demonstration Phase, the programme was rolled out in a staggered manner, with some of the 14 territorial NHS Boards and localities within NHS Boards starting before others, and some areas adopting elements of the programme before other elements. Thus Childsmile Practice began in 2006 in the West of Scotland, initially involving a limited number of dental practices and health visitor teams. Coverage was steadily extended within these NHS Boards, and other NHS Boards initiated the process in subsequent years. Several areas in the North of Scotland launched Childsmile Practice in 2007/2008, while the East of Scotland got underway with this element of the programme in 2009/2010. In the same manner, Childsmile Nursery and School began predominantly in the East of Scotland (but also in Ayrshire and Arran NHS Board) in 2006/2007, gradually increasing the number of participating educational establishments and expanding into other NHS Boards. While this process was necessary for logistical purposes, it had the advantage of enabling lessons from one area to inform rollout in others.

Monitoring Childsmile

As with other national health demonstration projects in Scotland, thorough and objective evaluation is central to Childsmile's role as a testing ground for national action, and resources are in place to support monitoring and evaluation, including embedded research projects. Monitoring - the continuous or periodic collection and analysis of data in order to compare actual and expected performance - is used to assess progress, check delivery of inputs, and provide a basis for corrective action. The monitoring arrangements described in this paper comprise one component of a comprehensive, multi-method evaluation strategy comprising both formative (to improve and refine the programme) and summative (to assess impact) methods.4,5,6,7 The evaluation strategy (to be discussed in a later paper) is designed to determine effectiveness, identify best practice, and inform decisions about future programming.

Monitoring of the Childsmile programme has four functions:

  1. i

    Management, including workforce planning and training

  2. ii

    Feedback of data on a service or area level over time to inform programme development and rollout

  3. iii

    Payment – to general dental practices which have signed on to the programme

  4. iv

    Evaluation – in relation to programme coverage, participation/drop-out, targeting, health service use, and health outcomes.

Two informatics tools are central to Childsmile monitoring and evaluation. Firstly, use of Community Health Index (CHI) numbers will permit linkage of data from a variety of sources, in support of the aim of establishing a longitudinal prevention programme, based on individual child-based oral health records of all contacts, including preventive treatment, advice-giving, check-ups, and restorative treatment. The NHS National Services Scotland Privacy Advisory Committee has approved the use of the CHI number to record-link Childsmile databases to the following key evaluation outcome measures:

  • Oral health outcomes via National Dental Inspection Programme (NDIP) data8

  • Dental service outcomes via registration and treatment data from the general dental service

  • Health service outcomes via data on hospital admissions for dental extractions

  • General health outcomes via BMI (body mass index) data from school health systems.

Secondly, use of the postcode-based Scottish Index of Multiple Deprivation (SIMD)9 allows examination of the extent to which the programme addresses and reduces health inequalities. SIMD is the key indicator for effective targeting of policies and funding to small area (700 population approx.) concentrations of deprivation in Scotland. It includes 37 indicators in seven domains. SIMD ranks are divided into quintiles, with quintile 1 covering the most deprived 20% of Scottish postcode areas. Being an indirect proxy for the deprivation experience of individual families, it carries the danger of 'the ecological fallacy' - in this case the assumption that residence in a deprived area demonstrates individual deprivation, and residence outside such areas indicates freedom from deprivation. With this reservation, the use of SIMD allows consistent monitoring of the targeting of Childsmile interventions towards families likely to be deprived.

Monitoring agencies

A number of universities and NHS agencies and boards from across Scotland collaborate in this work:

  • The Central Evaluation Research Team (CERT), based in the Community Oral Health Section, University of Glasgow Dental School lead on the overall evaluation of Childsmile. Monitoring data from Childsmile Practice is also collated here – including caries risk assessment forms and information for payments to practices. CERT also leads and coordinates research activity undertaken by Childsmile regional researchers based in NHS Boards and covering the North, East, and West of Scotland

  • The Health Informatics Centre (HIC), working with the Dental Health Services & Research Unit, University of Dundee, records Childsmile Nursery and School fluoride varnish application data using a web-based secure system which uploads data from the field via laptop computers on a daily basis

  • The Dental Informatics Programme based in the Information Services Division (ISD) of NHS National Services Scotland, processes GP17 forms used to pay practitioners as well as providing data for monitoring and evaluation of Childsmile Practice. ISD coordinates data from NHS Boards, benchmarking progress with the Childsmile Core programme.

Regular monitoring reports collating data from these sources provide descriptive statistics on the implementation and delivery of all four elements of the Childsmile programme.

Childsmile monitoring data: 2006-2009

The following sections of this article present monitoring data for the four Childsmile elements. Because of the staggered nature of the rollout described above, most of the data relating to Childsmile Practice refer to the West of Scotland, while those on Childsmile Nursery and Childsmile School refer mainly to the East.

Table 1 summarises Childsmile workforce and service provision at the end of the Demonstration Phase.

Table 1 Childsmile workforce and service provision by mid-2009
Full size table

Childsmile Practice

In the Demonstration Phase of Childsmile family involvement was initiated by health visitors completing a caries risk assessment (CRA) during home visits to the families of all newborn infants. Families identified by the CRA could access local Childsmile dental services, receiving oral health promotion and clinical preventive care tailored to their child's needs, supported by community-based Dental Health Support Workers. Thus for many children the CRA was the first crucial step in establishing an Integrated Care Pathway from birth through childhood, with appropriate care pathways for children with different levels of need. In 2009 the CRA was replaced by universal enrolment, but with the intensity of input to families based on their assessed need.

Table 2 shows assessment and referral totals in the West of Scotland for each half-year from mid-2006 to mid-2009. It also shows the number of practice first visits. Although these are records of contacts rather than individual children, they give a reasonably accurate picture of the level of engagement in Childsmile Practice over this period.

Table 2 Childsmile Practice: assessment, referral and practice contacts - all NHS boards
Full size table

Overall numbers grew steadily from 2006. By mid-2008 over 300 health visitors in the West of Scotland were completing around 800 CRAs of infants per month, and referring about 500 to primary care dental service settings via the Dental Health Support Workers. First practice visits increased five-fold between 2006 and 2009, partly due to the growth in the number of direct enrolments. As note a) to Table 2 explains, time lags between visits make it misleading to read row three as a percentage of rows one and two. It is more realistic to read diagonally down to the right from cells in row two. This suggests a referral rate of 50 to 60%, and attendance rate of 70 to 80% for first visits.

There is evidence that this activity was successfully targeted towards families living in areas of higher social deprivation. Births in the most deprived areas (SIMD 1) comprised 37% of all 2007 live births in the West of Scotland Heath Boards where Childsmile was active at that time (Greater Glasgow, Ayrshire and Arran and Lanarkshire). Births in SIMD 1 areas represented 45% of all CRAs completed by health visitors in those areas from 2006 to 2009, 55% of all referrals to Childsmile, and 56% of all practice first visits.

Figure 1 shows the growth in numbers of Childsmile general dental practices and clinics in the West of Scotland and of Childsmile-enrolled under-fives with these practices. In mid-2008 Childsmile began offering dentists in these practices a fee for providing fluoride varnish treatment to Childsmile enrolled children. In the first year of operation 1,513 applications had been made under this provision.

Figure 1
figure 1

Number of Childsmile practices and children (N) under 5 enrolled, West of Scotland, 2006-2009

Full size image

Childsmile Nursery and Childsmile School

By the second half of 2009 Childsmile Nursery and Childsmile School had expanded from their original East of Scotland base to cover 255 nursery schools and 200 primary schools in eight NHS Boards. Teams supporting these schools were manned by 32.5 whole time equivalent Dental Health Support Workers and 35 Childsmile-trained Extended Duty Dental Nurses (Table 1). Monitoring data reported here includes consent information and focuses on the twice yearly fluoride varnish application in targeted nurseries and schools.

Figure 2 illustrates the expansion of fluoride varnish applications in nursery and primary schools across the eight NHS Boards participating by mid-2009. The number of treatments totalled 27,924 over the two-and-a-half year period.

Figure 2
figure 2

Number of Childsmile fluoride varnish treatments in nursery and primary schools by school session and NHS Board, 2006-2009

Full size image

Figure 3 shows the number of consented and treated children as a percentage of the P1 and P2 school rolls in these NHS boards. The percentage of consented children who received varnish treatment was consistently above 80% both over time and across NHS Boards. However, there was considerable variation in consent rates: overall coverage was 47% of the P1 and P2 school rolls in these areas over the period.

Figure 3
figure 3

Percentage of children on school rolls in Childsmile fluoride varnish programme by participating Health Board, 2006-2009

Full size image

Childsmile Core

The two elements of Childsmile Core are: the offer of toothbrushing to all 3 and 4-year-old children attending nursery classes and to P1 and P2 children in targeted primary schools, and the distribution of toothpaste and toothbrush packs to every child in Scotland at six months (via the health visitor) and then via nurseries and targeted primary schools up to age six.

By the end of the 2008-2009 school year 95% of all Scottish nursery schools, attended by 97,000 3-5-year-olds, were participating in supervised toothbrushing.10 As Figure 4 shows, there was some variation between NHS boards.

Figure 4
figure 4

Percentage of nursery schools in Childsmile toothbrushing programme by Health Board, 2008-2009 school year

Full size image

Oral health pack distribution occurs twice yearly to the children in nursery schools. In 2008 pack distribution also began children in primary schools. NHS Boards report oral health packs are universally distributed to all nurseries (for 3-5-year-olds) and primary schools (for 5-year-olds).

Discussion

There are two general patterns which are apparent from the data presented in this paper: a consistent growth in activity over the three year period, and evidence regarding the extent to which Childsmile is reaching its target population. It is crucial to determine whether there is a social deprivation dimension to any shortfall in service delivery so that the programme can be adjusted to address this issue.11 More generally, the staggered roll-out of the programme complicates the interpretation and comparison of data across and within NHS Board areas.

Childsmile Practice

Figure 1 indicates the level of assessment and referral activity achieved within Childsmile Practice in the West of Scotland. Initial evidence suggests that a degree of targeting has been achieved by this programme of referral and enrolment. However, Table 2 shows the extent to which referral fails to result in a practice visit, and further analysis is underway to map the extent to which there may be a social deprivation aspect to this attrition.11

The Childsmile Practice monitoring database, holding information on approximately 35,000 children derived from paper based proformas, is dependant on complete and accurate manual completion of forms by health visitors, Dental Health Support Workers, and practices. Evidence of inconsistencies between areas, including failure to record contacts outside the practice ie at the family home, has prompted the simplification of record keeping and explicit management guidance regarding what is to be recorded by practices, health visitors and Dental Health Support Workers.

Childsmile Nursery and Childsmile School

As Figure 2 shows, there was considerable growth in fluoride varnish activity in nurseries and schools, and by mid-2009 almost 28,000 applications had been completed. While there was good coverage of those children who were consented, significant numbers were not consented for varnish application (it is estimated that only 2-3% are ineligible because of contraindications/adverse medical histories). Although consent tends to be higher in nurseries, this is counterbalanced by more absences on the day. Different agencies, including NHS Boards, have different approaches to obtaining consent, and to returning to treat children absent from the scheduled treatment session. As with Childsmile Practice, an exploration of any potential link between social deprivation and lack of take-up will be conducted.

The data presented in Figures 3 and 4 do not show the extent to which individual children received a complete course of applications during the time they attend nursery (ie three to four six-monthly applications, consistent with the twice-yearly frequency recommended by practice guidelines (eg the Scottish Intercollegiate Guideline Network guideline 832). Such an analysis requires a retrospective analysis of children first treated two years earlier, which currently limits the numbers involved, and may not reflect current coverage, as teams, nursery schools etc. became more practised in the delivery of programme. Further work on the extent to which varnish courses are completed is currently underway and will be reported in a future paper.

Development of the bespoke HIC/DHS&RU system for Childsmile Nursery and Childsmile School seeks to integrate some programme management tasks and electronic prescribing within the 'monitoring' system and to facilitate the Childsmile teams working more efficiently by communicating electronically from different locations.

Childsmile Core

By 2009 almost all 3 and 4-year olds attending nursery could take part in daily supervised toothbrushing (Fig. 4). They also received oral health packs, alongside an increasing number of primary school children. However, there is no reliable way of distinguishing how many nursery school children actually take part in toothbrushing on a regular daily basis, if sessions always take place, or if packs are actually distributed to (or used by) individual families. Most NHS boards only count orders for packs, not receipt by families, and vary in their monitoring arrangements.

Conclusion

The instrumental use of monitoring data through feedback to health service staff has been ongoing since mid-2007, enabling staff to reconfigure aspects of the programme to better achieve its aims. For example, analysis of CRA monitoring data indicated shortfalls in the referral of at risk infants. As a consequence the assessment process has been simplified and Dental Health Support Workers integrated within the public health nurse team structure in the West of Scotland. More generally, Childsmile will increasingly become embedded in overall child health services, including pre-school child health surveillance, as it develops beyond its development stage, allowing the simplification, streamlining and speeding up of referral, information sharing and monitoring. Other issues include attrition in numbers completing repeat attendances at Childsmile practices; non-consent or non-attendance for varnish treatment; and a fall-off in toothbrushing after the first year at primary school.

Feedback of monitoring data promotes joint-working between programme implementers and the CERT to further explore observed patterns through mixed-methods studies such as that relating to the communication strategy report commissioned from the Institute for Social Marketing,12 and described in the companion paper to this.1

As pointed out at the beginning of this paper, all these measures relate to the process of delivery of the Childsmile programme. In order to obtain robust and comprehensive outcome measures of child oral health, the next stage is to identify groups of families based on their initial and continuing contact with Childsmile Practice, including direct referrals as well as CRA-based referrals. This material will use CHI numbers to enable cross reference with ISD records on registrations and treatment history. These data will permit a more detailed examination of the extent Childsmile has successfully reached the most at-risk groups, using SIMD as a proxy for deprivation. Ultimately Childsmile records will also link to National Dental Inspection Programme (NDIP) records of the dental status of all 5 and 11-year-olds. These linkages will greatly enhance the detailed evaluation of the positive impact of the Childsmile programme on the oral health of Scotland's children, the factors and circumstances which promote this end, and the overall cost-effectiveness of the programme.