The British Paediatric Surveillance Unit (BPSU) conducts surveillance of rare paediatric conditions using active, or prospective, case finding. The reliability of estimates of incidence, which is the primary outcome of public health importance, depends on ascertainment being as near complete as possible. This paper reviews evidence of the completeness of ascertainment in recent surveillance studies run through the BPSU. Ascertainment varied between 49% and 94% depending on the study. These are upper estimates. This was the basis of a discussion on barriers and facilitators of ascertainment which we have separated into factors related to the condition, factors related to the study methods, factors related to the study team and factors related to the surveillance system infrastructure. This leads to a series of recommendations to ensure continuing high levels of ascertainment in active surveillance studies.
- general paediatrics
- disease surveillance
- case ascertainment
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
What is already known on this topic?
Active disease surveillance depends on high case ascertainment.
The British Paediatric Surveillance Unit methods were designed to give high case ascertainment when setup in the 1980s.
Changing working patterns, technology and the healthcare environment may have affected case ascertainment in positive and negative ways.
What this study adds?
Ascertainment levels remain high.
Factors related to the condition under study, the study methods, the study team and the surveillance system can all affect ascertainment.
There are ways to optimise ascertainment by addressing these factors in study setup and surveillance system methods.
The British Paediatric Surveillance Unit (BPSU) is one of a number of surveillance systems (eg, British Ophthalmological Surveillance Unit—BOSU, Child and Adolescent Psychiatric Surveillance System—CAPSS and members of the International Network of Paediatric Surveillance Units—INoPSU) which use active, or prospective, case ascertainment in order to measure incidence. In 1986, the BPSU introduced the method of monthly cards sent to every consultant paediatrician in the UK and Ireland as a simple yet robust method of ascertaining incident cases of a limited number of uncommon conditions. Since its inception over 120 surveillance studies have been run through the BPSU with many important impacts on public health and clinical practice.1 However, the ability to ascertain cases may have been affected by changes in clinical working patterns such as subspecialisation and less continuity of care; developments in BPSU methods particularly use of electronic communication; and changes in the regulatory environment surrounding research and data privacy. The potential impact of a surveillance studies often depends on the reliability of the estimate of incidence. For rare conditions, this is sensitive to small changes in the number of cases which are ascertained. In this paper, we first review available evidence on the levels of ascertainment in recent studies, next we describe barriers and facilitators, and go on to recommend measures to optimise ascertainment.
The surveillance system used by the BPSU is illustrated in figure 1. Every month all substantive paediatricians (ie, consultant and associate specialists) are requested to report whether they have seen a new case of any of the conditions currently under surveillance. In order to measure compliance, they are also requested to report if they have seen no cases. Initially postcards were sent and returned by mail but in 2011 electronic reporting was introduced and postcards were completely phased out by 2015. The study team for each condition is notified of all case reports and issues a clinical questionnaire to the reporting paediatrician. The studies are run by research teams independent of the BPSU.
Lists of current postholders are updated from multiple sources, for example, Royal College of Paediatrics and Child Health member lists, recipients of certificates of completion of training, self-registration from college members.
This paper is based on data presented and discussed at a meeting on ascertainment hosted by the BPSU in June 2017. Participants were invited if they were involved in surveillance unit management (including other units such as BOSU and CAPSS) or they had run studies through the BPSU or other paediatric surveillance units. We selected particularly those involved in multiple studies or long-running studies. We have listed those attending in the authorship list. A number of the participants are reporting paediatricians themselves.
The discussion between workshop participants was structured round a series of questions which we have summarised into themes. The agenda for the meeting, including formal presentations and discussion questions, is listed in web supplementary appendix 1.
What is the scale of (under)ascertainment
Completeness of reporting
Over 3500 e-cards are sent each month. The monthly average response rate of over 90% did not change over the period when electronic reporting was introduced (see table in web supplementary appendix 2). Return rates of clinical questionnaires range from 80% to 95% between studies.
Multiple sources of ascertainment
Using multiple sources for ascertainment is a means for both monitoring and improving ascertainment. A review in 2006 found that 38/59 (64%) of BPSU studies used additional sources, with some studies using several sources.2 More recent data are shown in table 1.
Studies using additional reporting sources include the study on elevated lead in children which ascertained cases from the BPSU, the Supraregional Assay Service Trace Elements laboratories and the health protection teams of the five national public health organisations.3 All of these sources should have had complete ascertainment. Seventy per cent was obtained through the BPSU, 65% through the laboratories and 40% through the public health organisations.
Several studies have used multiple data sources because paediatricians would not be expected to know all cases and hence BPSU ascertainment would be low. These include the National Surveillance of HIV in Pregnancy and Childhood (NSHPC) which uses parallel reporting systems to collect data from obstetric and paediatric units throughout UK and Ireland as well as the BPSU (http://www.ucl.ac.uk/nshpc). Long-established networks and the ability to link data about cases have resulted in a high level of ascertainment. The study on severe visual impairment was run by the BPSU and BOSU, while a study on young people with attention-deficit hyperactivity disorder in transition to adult services (CATCh-uS) was run with CAPSS.4 5
Comparison with lab-based infectious disease surveillance data
A number of studies of infectious disease or their complications have also used lab-based databases. These record bacterial isolates but few clinical details. Lab-based data can be used to backtrack to clinicians for clinical data if appropriate permissions and data confidentiality arrangements are complied with.
The study on invasive group B streptococcal disease in infants found around 50% were picked up by the BPSU reporting system.6 The remaining cases were reported through a network of lab-based systems covering the five countries. However, after backtracking and intensive case finding, it was possible to obtain clinical information from reporting paediatricians on 84% of cases.
Comparisons with lab-based non-infection screening or biochemistry data
In a study on congenital adrenal hyperplasia with both lab-based and BPSU case notifications, only 8 out of 144 cases were not reported through the BPSU. This study actively sought cases by backtracking from lab reports from the 12 biochemistry laboratories that carry out the assays in the countries covered by the study.9
In contrast, a study by the same team on congenital hypothyroidism had lower initial ascertainment (see figure 2).10 After removal of errors and duplicates, 427 cases were reported to the BPSU through the electronic reporting card system. In 67 of these, a clinical questionnaire was not submitted (ie, the questionnaire return rate was 84%). At the same time, there were 704 cases reported through the labs with abnormal screening results. Twenty-eight of these were not able to be traced further. Thus, the initial ascertainment rate through the BPSU would have been 52% (ie, (22+338)/(22+338 + 338)). Further intensive case finding and, in some cases, collecting only minimal data resulted in useful data being available on 698 of 739 (94%) case reports. The remaining 41 cases with no clinical data were 13 BPSU reports where no questionnaire was completed and 28 lab reports with no traceable clinician.
Capture–recapture analysis is a means of estimating a total population from two independent samples so potentially could be used to measure the true incidence of a condition where there is under-ascertainment in the different data sources. A review in 2006 showed that 6 out of 38 BPSU studies which used more than one data source carried out capture–recapture analyses.2 A more recent study of vaccine associated intussusception used the BPSU and Hospital Episode Statistics as separate data sources.11 In total, 200 cases were confirmed, 163 via the BPSU (ie, ascertainment calculated at 86%). When subjected to capture–recapture analysis a presumed total population of 233 cases was estimated (giving a final BPSU ascertainment rate of 70%).
Capture–recapture analyses depend on six basic assumptions of which two, homogeneity and independence, are rarely met in BPSU studies (see table 2). Failure to meet these criteria makes capture–recapture analysis unsuitable for estimating true ascertainment.
What factors are barriers and facilitators to ascertainment?
Factors related to the condition under study
Clinical and public health importance
Paediatricians are more likely to recall and report conditions in which the clinical relevance or public health importance is clear and relates to their day-to-day work. Put simply, paediatricians must find the condition noteworthy.
Conditions need to be recognised by reporting paediatricians, many of whom may be unfamiliar with the specific condition. This requires a precise case definition. The surveillance case definition needs to be highly sensitive to ensure that all cases are recognised and reported. The study team then apply an analytic case definition which needs to be highly specific.
The burden on reporting paediatricians needs to be manageable. Studies with a high reporting burden, particularly when this falls on a limited number of clinicians or subspecialties, have found that ascertainment may suffer. Some studies have resolved this by specific arrangements for data collection with high reporting units. Studies which are likely to place a heavy burden on reporting clinicians are requested to propose a procedure to alleviate this burden.
Concerns over data disclosure and confidentiality
Some clinicians have been reluctant to report cases because they have concerns about data confidentiality or disclosure of identifying information, particularly when the condition under study is sensitive or has a high public profile. There have been misunderstandings by local research and development (R&D) departments about which data may be reported and released. All these issues though are covered by the governance approvals required before surveillance commences.
Factors related to the study methods
Multiple sources of data should always be considered unless the condition is exclusively seen by paediatricians (or the reporting clinicians for other surveillance systems). These data sources may be used in different ways; as alternative sources of ascertainment, as a complement to primary surveillance, for example, by backtracking from lab reports to trace clinicians who may not have reported the case initially, or, rarely, as independent data sources for capture–recapture analyses.
Clinical questionnaires are more likely to be completed if the questionnaire is concise and the questions are focused on answering the study objectives. Questionnaire length needs to be limited and the study objectives restricted to those with clinical or public health relevance. A detailed data analysis plan is an effective means of ensuring that each item in the questionnaire will provide useful and analysable information (see https://www.rcpch.ac.uk/resources/applying-bpsu-run-study-orange-card-reporting-scheme for advice on producing a data analysis plan).
Where initial questionnaires were not completed, even with reminders, the congenital hypothyroidism study used a reduced questionnaire with limited data enabling partial analysis of some key study objectives (eg, to exclude false positive reports from screening labs and to confirm initial treatment).10
Factors related to the study team
Engagement of reporting paediatricians
Study teams can enhance interest by providing feedback to reporting clinicians and dissemination of findings in conferences and academic publications. For example, the Progressive Intellectual and Neurological Deterioration study offers diagnostic support to clinicians reporting a case, the NSHPC fosters links with respondents by regular feedback and updates on study news and publications, and the NSHPC and the CATCh-uS studies offer certificates for completion of study questionnaires.
Personal contact with reporting clinicians or their secretaries may be more effective than letters, or emails. Clinical questionnaires should be dispatched within days of the case being reported and queries from reporting clinicians responded to rapidly. An offer to complete a clinical questionnaire over the phone is more likely to result in useful data than simply resending the questionnaire. These approaches would all be permitted under the research governance, ethical and confidentiality regulations.
The administrative burden of a study is often underestimated. Those studies which have committed enough administrative support have generally had higher rates of completed clinical questionnaires, which will result in more accurate ascertainment.
Factors related to the surveillance system infrastructure
Maintaining the reporting database
Return of the monthly report card from clinicians >90% and of clinical questionnaires >80% are minimal rates accepted by the BPSU. If these fall lower then active steps of ensuring accuracy of the database of clinicians, and chasing up individual clinicians are necessary. Return rates of the monthly card increased after electronic reporting was introduced (table 1).
Although the response to the electronic monthly report remains high, there is no facility for clinicians who report a case being able to record a local identifier (eg, hospital number) with the case report. Study teams and reporting clinicians both feel this to be a significant weakness of the move to electronic reporting. This has now been addressed by sending an aide memoir straight back to the clinician after reporting a case. In addition, study teams which use an electronic questionnaire can reduce the time period between reporting a case and completing a questionnaire.
Electronic clinical questionnaires
Electronic forms for online data collection have been found to be efficient and acceptable to respondents. The BPSU currently recommends electronic data collection, particularly where data are transmitted and stored securely. Eventually we aim to establish a data platform to allow secure processing as well as the collection of all data through one portal (a ‘safe haven’).
Dealing with the changing regulatory environment
Changing research governance requirements, ethical frameworks and confidentiality and data privacy arrangements within the five countries covered by the BPSU have created a complex regulatory environment which in some cases has resulted in apparent incompatibilities. This has threatened the continuing operation of surveillance as a whole, but even within the limited aspects of case ascertainment, data privacy arrangements which prohibit potentially identifiable data from leaving the country it was collected in have jeopardised identification of duplicate reports of the same case.
Estimates of ascertainment from recent BPSU-associated studies range from 49% to 94%. The changing technological, regulatory and health service environments have presented opportunities to improve ascertainment and data accuracy, but also thrown up new barriers which could not have been predicted when the systems were setup. Nevertheless, the bulk of work undertaken by the surveillance systems still revolves around ensuring a high response rate of the monthly reports, ensuring clinicians complete questionnaires on cases they have reported, and developing and refining case definitions during the setting up of surveillance studies.
Limitations and generalisability
Estimates of ascertainment depend on comparison of imperfectly ascertained sources of data. The true level may be lower than we have calculated. However, capture–recapture analyses, where justified, support our estimates. The facilitators and barriers to ascertainment are based on opinions of an informed group but are not subject to any confirmatory evidence.
The factors which influence ascertainment depend on the methods of the surveillance system. They may not be applicable to other surveillance units either in the UK or internationally. However, we included members of other UK surveillance units in our group and personal communications within INoPSU suggests similar issues are recognised elsewhere.
Recommendations arising from the group are collated in the table 3 which will be used in assessing future BPSU studies and which may be useful for other surveillance systems in the UK and worldwide. Other types of health service research and evaluation may also find some of these recommendations helpful.
Case ascertainment is always likely to be incomplete so reported estimates of incidence would be better described as ‘minimal incidence’. We continue to recommend the use of complementary data sources wherever possible. Using these to support intensive case finding and backtracking is a more productive approach than simply using them as alternative sources of ascertainment. Surveillance of rare conditions through the BPSU is a success story of modern paediatrics1 and remains as relevant now as before. It is essential to continue monitoring ascertainment to demonstrate the robustness of the system and to justify the continued participation of paediatricians and healthcare organisations in surveillance of rare childhood conditions.
Contributors The two named authors collected the data from the ascertainment group, carried out the analysis and drafted the paper. The members of the BPSU Ascertainment Group participated in the meeting, contributed data and have reviewed versions of the manuscript. They have been asked to confirm their agreement to have their names published as collaborators.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data are available publicly as shown in references or have been supplied by contributors and are acknowledged. Further access to the data if required can be supplied by applying to either of the named authors.
Collaborators Members of the BPSU Ascertainment Group: Adams N—Public Health England. Avis J—British Paediatric Surveillance Unit. Barker L—Leeds General Infirmary. Bedford H—Institute of Child Health, University College London. Eke H—University of Exeter Medical School. Foot B—British Ophthalmology Surveillance Unit. Francis K—National Surveillance of HIV in Pregnancy and Childhood. Hunter L—Royal College of Paediatrics and Child Health. Irwin A—British Paediatric Allergy Immunology and Infectious Disease Group . Janssens A—University of Exeter Medical School. Knowles R—Institute of Child Health, University College London. Ladhani S—Public Health England. Lynn RM—British Paediatric Surveillance Unit. Michie C—Royal National Orthopaedic Hospital. Oeser C—Public Health England. Peters H—National Surveillance of HIV in Pregnancy and Childhood. Philbin M—Royal College of Paediatrics and Child Health. Powell A—Addenbrookes Hospital, Cambridge. Reading R—British Paediatric Surveillance Unit. Samad L—Institute of Child Health, University College London. Simms I—Public Health England. Simpson M—Royal College of Paediatrics and Child Health. Stevens S—National Congenital Anomalies and Rare Disease Registration Service. Tookey P—Institute of Child Health, University College London. Verity C—Addenbrookes Hospital, Cambridge. te Water Naude J—Welsh Paediatric Surveillance Unit. Winstone AM—Addenbrookes Hospital, Cambridge. Wood R—NHS National Services Scotland, Information Services Division.
Patient consent for publication Not required.