Results

Seven hundred and sixty six (80%) clinicians on the mailing list, 48 investigators, and 15 public health professionals responded to the survey questionnaires.

PUBLIC HEALTH IMPORTANCE

Conditions are only accepted for study through the APSU if they are uncommon (and hence require national data collection to provide meaningful information), of public health importance, and are seen by those on the mailing list. Data should not be available from another source. Some are associated with substantial morbidity or mortality and/or result in high utilisation of health resources, while others are communicable or preventable.

THE SYSTEM

The mailing list, which is updated at least annually, can be adapted to target a broader group of respondents relevant to conditions listed—for example, child psychiatrists were included during studies on Rett syndrome and childhood dementia. Poor responders are also reviewed periodically.

USEFULNESS

Does the system detect trends signalling changes in the occurrence of disease?

The APSU can monitor trends in disease incidence, management, and outcome over time by monitoring a disease over several years or by performing cross sectional studies separated in time. Case identification is not usually sufficiently timely to allow detection of disease outbreaks or epidemics as they occur because of the inherent delay in receiving notifications (report cards are sent at the end of each month and their return time is variable). However, by providing background information on haemolytic uraemic syndrome cases occurring nationally, the system facilitated the investigation of an epidemic in South Australia that occurred after the condition had been listed on the monthly card.5

Does the system provide estimates of the magnitude of morbidity and mortality related to the health problem under surveillance?

Clinical information collected through the APSU enables detailed assessment of acute and chronic morbidity associated with conditions studied. Such information was rarely available previously. For example, congenital varicella was thought to be extremely rare in Australia. However, the APSU study has provided an estimated incidence of 0.8 in 100 000 births, similar to recent estimates from the UK.6

Although the APSU is not suitable as a sole source of mortality data, it can provide information on conditions with acute mortality (Kawasaki disease, congenital rubella, haemolytic uraemic syndrome), or where follow up studies of a cohort have been undertaken (extrahepatic biliary atresia, Rett syndrome).

Does the system stimulate epidemiological research likely to lead to control or prevention?

Preliminary results of the drowning/near drowning study showed differences between states, prompting development of state registers that will enable case control studies to examine prevention strategies.

The Rett syndrome study provided cohorts used for further research into associated skeletal abnormalities7 and for a randomised clinical trial of treatment.

Does the system identify risk factors associated with disease occurrence and/or lead to identification of prevention strategies?

No case control studies to examine risk factors have yet been conducted through the APSU. Risk factors and prevention strategies have, however, been implicated in a variety of ways.

The extrahepatic biliary atresia study, using internal controls, confirmed late diagnosis was a risk factor for poor outcome. The need to educate health workers and parents to facilitate early diagnosis was highlighted as an important prevention strategy.

The haemolytic uraemic syndrome study identified several organisms, previously not implicated in Australia. Data contributed to the development of strategies for disease control and prevention, including: (1) changes to the code for the manufacture of fermented meat products; (2) requirement for notification of haemolytic uraemic syndrome cases to some state public health departments; (3) and public education about food storage and preparation.

Comparison with population data showed that children born in Australia in 1995 and 1996, whose mothers were born outside Australia, were at increased risk of congenital rubella. Establishing the vaccination status of this subgroup of women might prove to be an important preventive strategy.

Identification of cases exposed perinatally to human immunodeficiency virus (HIV)/AIDS has enabled a review of the uptake and efficacy of zidovudine treatment.

Does the system lead to improved clinical practice by health care providers who are the constituents of the surveillance system?

Of the 766 clinicians who responded to the questionnaire, 33% said information provided by the APSU had informed or changed their clinical practice; 62% found definitions and other information provided to be educationally useful, 52% had read APSU related publications, and 44% had seen conference presentations. Sixty eight per cent had read the last annual report and 70% of those found it useful. To date, publications containing APSU data or describing the system include 18 peer reviewed articles, one book chapter, two Masters of Public Health treatises, and seven annotations.

Has the system led to changes in public health policy?

Indirect measures of changes in public health policy were available from questionnaire responses. Thirty nine of 48 investigators said their study allowed evaluation of current medical management or policy, informed future medical management or policy, and/or contributed to prevention policy.

Of the 15 public health professionals who responded to the questionnaires, 12 had heard of the APSU. Two had received and read the annual report and both found it useful. Three had used information derived from APSU to formulate policies and best practice guidelines, to recommend future research, and/or to identify preventive strategies.

Has the APSU provided a mechanism for national collaborative research?

Over 50 investigators were involved in the 15 studies conducted to the end of 1996. Of these studies, 13 were multicentre and nine involved more than one state or territory. A wide variety of scientific disciplines were represented. Ninety per cent of investigators said their research could not have been conducted without national case ascertainment; 83% said this could not have been achieved without the APSU, and 90% felt that “in retrospect, the APSU was the best way to conduct their study”.

SIMPLICITY

The system is simple (fig 1). Report cards are easy to complete, requiring the clinician only to indicate the number of cases they have seen in the previous month or to tick the “nothing to report” box. Cards are reply paid (fig 2).

The APSU related workload for most clinicians is minimal. Of 766 clinicians, 670 (88%) said they did not find it a burden to complete and return the monthly cards. By the end of 1996, nearly half had never reported a case, and a third had reported only one or two cases. The workload for clinicians reporting cases depended on the complexity of the study questionnaire. The proportion of clinicians who reported that questionnaires were “too long” ranged from none in the study of subacute sclerosing panencephalitis to 16% in the study of drowning/near drowning. The proportion who thought information requested was “hard to find” ranged from none in the study of severe combined immunodeficiency disorder to 39% in the study of acute flaccid paralysis.

FLEXIBILITY AND TIMELINESS

A new condition can be listed for surveillance within one month, the interval between mailings. However, a mailing to request notifications could be performed within days in the case of an epidemiological emergency. The interval from receipt of a non-urgent study application to the condition appearing on the report card varied from three to 12 months. This is influenced by the requirement for a scientific review process, the time taken for modification required by reviewers, and the availability of space on the card. Case definitions of conditions can be changed within one month.

Ninety per cent of clinicians said they would report by telephone or fax if an important public health reason was provided. During the haemolytic uraemic syndrome epidemic, which occurred while this condition was under surveillance by the APSU, the system was sufficiently flexible to allow clinicians to make rapid contact by telephone. However, in the acute flaccid paralysis study (where telephone/fax notification is requested so clinicians can be reminded of the requirement for stool testing for poliovirus), the number of notifications received by telephone/fax (32%) rather than by card has been disappointing (Herceg A, personal communication). Reporting by electronic mail commenced in February 1997 and was used by 125 (14%) reporting clinicians by May 1998. It is anticipated that this will increase as more clinicians gain access to email.

The length of time between notification of a case to the APSU, receipt of a questionnaire by the clinician, and receipt of a completed questionnaire by the investigator is important. We did not collect this data systematically, but 15 clinicians reported substantial time delay (up to seven months) between notification of a case to the APSU and receipt of a questionnaire from the investigator.

ACCEPTABILITY

Participation in APSU activities was used as an indicator of its acceptability. The average monthly card return rate increased from 88% in 1993 to 93% in 1996. Forty per cent said they would not return the card if it was not reply paid. The median return rate of study questionnaires was 92% (range, 59–100%). Between May 1993 and December 1996, 18 applications for studies were received.

Ninety two per cent of responding clinicians thought the number of conditions monitored was appropriate or could be increased; 67% reported “nearly always” or “always” reading study protocol sheets, while 5% “never” read them. Thirty two per cent of clinicians provided comments in response to an open ended question asking for their opinion of the APSU and suggested areas for improvement (table 2). Requests for the provision of specimens diminished the acceptability of some APSU studies. Nine of 41 clinicians notifying cases of acute flaccid paralysis and 16 of 56 clinicians notifying cases of haemolytic uraemic syndrome said they objected to obtaining stool samples.

SENSITIVITY

As an indirect indication of sensitivity, 5% of clinicians on the mailing list reported returning cards with “nothing to report”, despite knowledge of a case, assuming that other clinicians caring for the child would notify. In addition, 1% said that they had failed to return a card despite knowledge of a case.

To estimate the sensitivity of the APSU, cases were ascertained from alternative sources for five studies (table 3). Sensitivity ranged from 31% (drowning/near drowning) to 92% (congenital rubella).8

POSITIVE PREDICTIVE VALUE

Positive predictive value is defined as the proportion of notified cases who actually do have the condition under surveillance.2 Table 4 shows all cases reported to the APSU since 1993 and their status at the end of 1996. Positive predictive value2 can be calculated for the APSU in three ways. First, all duplicates and errors could be considered invalid and used in the denominator. However, the APSU encourages duplicate notification to maximise case ascertainment. Thus, alternative methods of calculating positive predictive value can either exclude duplicate reports or include them in the numerator. The use of these calculations produces quite different estimates of positive predictive value, as shown in table 4. Using the latter two methods, most notifications had a positive predictive value above 70%.

REPRESENTATIVENESS

Representativeness2 9 of the APSU could not be assessed directly because alternative sources of national case ascertainment do not exist for many conditions or are not comparable. There was minimal variation between response rates by state, territory (range, 90–97%), or specialty group (range, 90–94%) in 1996. This is an indirect indicator of representativeness.

RESOURCES USED TO OPERATE THE SYSTEM

The resources required to operate the APSU (the “direct costs”), comprise the personnel and financial resources expended in collecting, processing, analysing, and disseminating surveillance data. These currently total ∼ US$57 000/year (approximately £35 650), but this does not include salaries of the director and assistant director, or the costs incurred by investigators. Paid personnel include three part time administrators/data managers. There is currently no charge for investigators.

Discussion

Although the APSU fulfils most of its stated objectives and meets CDC guidelines salient to these, we identified areas where the operation of the system could be improved. We also identified methodological problems in applying CDC criteria to this type of surveillance unit.

The operation of the APSU is simple, acceptable to the mailing list, and sufficiently flexible to allow surveillance of acute conditions of public health importance. The usefulness of the APSU for monitoring such conditions pertains both to the system as a whole (encouraging national collaboration, improving clinical practice) and to specific studies. Although it was not designed to identify contacts or sources of infection of acute cases, and is not sufficiently timely to do so, the APSU is an important adjunct to the public health response. However, dissemination of information to public health professionals was identified as suboptimal. This and other limitations identified by this evaluation have resulted in the recommendations shown in fig 5.

The attributes of direct relevance to the fulfilment of the APSU’s primary objective, namely the sensitivity and representativeness of the system, were the most difficult to evaluate. One reason for this is that for studies accepted by the APSU there should be no alternative source of national case ascertainment. To calculate sensitivity, we used either national (Rett syndrome association,10Australian Hearing Services), or state based sources (NSW inpatient data, NSW mandatory reporting). For most conditions, comparative data were only available for part of the surveillance period and were obtained by “passive” rather than “active” reporting.11-13 Limitations of using alternative sources include financial costs, privacy issues, delay in gaining access to data, variable definitions, different age cut offs, and the fact that hospital inpatient statistics databases do not provide data for outpatients.14

Despite these limitations, sensitivity estimates for some studies were achieved and were generally satisfactory. The calculated values may reflect issues of poor data comparability as much as limitations of the APSU.

In situations where the APSU has been used to supplement other sources of case ascertainment, as in the HIV/AIDS study,15 the sensitivity of the system does not need to be high. For future studies with a high sensitivity requirement, APSU will recommend the prospective use of alternative sources of case ascertainment, either as a check of sensitivity or as a supplementary source of cases. Several British Paediatric Surveillance Unit studies have reached similar conclusions.16 17 To overcome problems associated with sensitivity estimates, some systems have used complex statistical techniques, such as capture/recapture analysis, to estimate sensitivity.18 However, these depend on obtaining data from independent sources, which are often not available, and on the fulfilment of various assumptions, many of which are complex and untestable.19

A surveillance system that is representative accurately observes both the occurrence of a health event over time and its distribution by person and place in the population at any point in time.9Uniform participation of clinicians by geographical distribution and speciality suggests that the APSU is representative. However, because no directly comparable data sources exist for the review of “missed” cases, there may never be a satisfactory means of assessing this factor.9

The APSU encourages notification of duplicate and “probable” cases (not fulfilling complete case definition) to maximise case ascertainment and information obtained. However, this has implications for calculation and interpretation of positive predictive value. A positive predictive value that allows for the incorporation of information about probable or duplicate cases is the most meaningful indicator of the value of a notification.

The success of a surveillance system depends on an appropriate balance of characteristics to meet its specific requirements.2While explicit guidelines for the evaluation of surveillance systems have been developed by the CDC, not all of the suggested criteria are of equal importance, applicable to all components, or readily evaluable for the APSU. Equally, the appropriateness of the suggested measures in evaluating this type of surveillance unit was not straightforward.

Our evaluation has helped prioritise attributes such as sensitivity, which are directly relevant to the specific objectives of the APSU, while giving lesser weight to others that are less central to the achievement of those objectives.

Currently, the APSU’s direct operating costs are moderate and it could be argued that the system, which facilitates simultaneous research into several conditions, is cost effective.17 Indirect costs, costs of secondary data sources, and costs averted by surveillance (savings resulting from preventing morbidity through surveillance data) are difficult to identify and quantify. Methods for formal cost benefit analyses of surveillance systems need to be developed. Such information will not only enhance the completeness of any evaluation, but will be increasingly necessary to secure funding for the maintenance of the operation of such systems in the future.