METHODS

Full details of the development of the database, including search strategy, sources, and reliability assessment of the citation selection process, are reported elsewhere.¹⁰ A list of website addresses scanned for this study can be found in Appendix 2 of the Technical Report.¹²

According to Drummond et al, an economic evaluation is defined as “the comparative analysis of alternative courses of action in terms of both their costs and consequences”.¹³ For the PEDE Project, a publication was accepted as an economic evaluation only when a comparator existed and descriptions of both costs and health outcomes were present. The economic evaluation did not have to be the primary objective of the study to be eligible. The inclusion criteria required that the study contained original analysis and included the evaluation of an intervention such as a medical or surgical treatment, a programme, a service, or a new process; that the intervention was directed at the paediatric population including neonates, infants, children, or adolescents less than 19 years; that if interventions were aimed at pregnant women or mothers, outcomes were measured in offspring; that the comparator was either real or implied, such as in a pre- and post-intervention, or a ‘do nothing’ or ‘usual care’ approach; that a health outcome, intermediate or final, was reported; and that costs were measured and reported. Randomised controlled trials and observational studies were eligible, as were modelling studies and meta-analyses if they included novel data aggregation and new analyses. Studies were excluded if they were cost of illness or cost of prevention studies, if a specific intervention was not evaluated, or if they assessed family planning interventions related to birth control that included outcomes such as couple year of protection or number of averted births; however, if the intervention was related to perinatal screening and the outcome was the number of averted births with a malformation or disease, the study was eligible. Interventions consisting of a guideline, a continuous quality improvement process, or a new operating procedure or policy targeted toward improving practice or efficiency were excluded, as were studies where costs were not quantified. To restrict the database to original, full economic evaluations, abstracts from conference proceedings, methodological papers, papers without original analyses, policy statements, case studies or reports, letters, editorials, or notes were excluded.

The application of the inclusion and exclusion criteria resulted in 787 eligible citations between 1908–1999. The following data were collected from each eligible citation: journal, target population, ICD-9-CM disease class,¹⁴ age group, intervention being studied, primary outcome measure, and health economic analytical technique, including cost-effectiveness analysis (CEA), cost-benefit analysis (CBA), cost-utility analysis (CUA), and cost-minimisation analysis (CMA). Where more than one analytical technique was used, a primary technique was designated by the abstractor using the following order of importance: CUA>CBA>CEA>CMA. All interventions described in the citations were categorised as either prevention, treatment, programme (organisation or organisational unit, clinic, department, or health system), surgical, educational, healthcare delivery (process, service, tool, or treatment pathway), detection (tests used to screen for disease or the potential of developing disease), diagnosis (the use of clinical tests to confirm the cause of illness), or psychological. Data entry and data management were performed using Microsoft Access 97. In addition, the citation identification number was used to link the Access database record to its full citation, including MeSH terms and abstract, in an EndNote (version 3.0.1) bibliographic database.

Trends in the paediatric health economic evaluation methodology were investigated by performing one way frequency distributions and two way cross tabulations on the variables in the database, including publication year, outcome category, analytical technique, and journal type.

RESULTS

Volume of publications

While the annual number of publications was fairly constant from 1980 to 1989, the volume of publication grew dramatically during the period of 1990 to 1999. The volume and growth by 5 year period is displayed in table 1. The average 5 year growth rate was 96%. Over half of the papers in the database (440) were published between 1995 and 1999.

Outcome measures

As several hundred different outcome measures were reported in the literature, these were grouped into categories as indicated in table 2. The most common category of outcome measure over the study period remained observed cases of disease/condition/abnormality. The proportions of papers studying cases of death and cases of cures/improvements/healing declined slightly over time, while changes in physiological measures and cases of complications/adverse events became more common as outcome measures.

A cross tabulation of intervention type by outcome category demonstrated that 58% of studies of health prevention interventions used an outcome categorised as cases of disease/condition/abnormality. Studies of health treatments commonly used cases of disease/condition/abnormality and cases of cures/improvements as outcome measures. Of the economic evaluations of paediatric health programmes, 74% were fairly evenly distributed across the outcome categories of cases of disease/condition/abnormality, cases of death and changes in physiological measures.

Analytical technique

Depending on the way health outcomes are measured and reported, economic evaluations are typically classified as a CEA, CUA, CBA, or CMA.¹³ In CEA, consequences are measured in natural units, such as cases of disease detected or life years gained. In CBA, the outcome is monetarised, so that costs associated with resource use and administration of the intervention are subtracted from the monetary value of the health benefit achieved. In CUA, the outcome, traditionally life years gained, is weighted by the patient’s preference for a particular health state, resulting in a quality adjusted life year (QALY). In CMA, the outcomes associated with the intervention and the comparator are comparable and only costs are considered. Marked changes in the type of analytical technique used over time was evident, as seen in table 3. While CEA was the most common technique used, accounting for the majority of evaluations in all time intervals, the proportion of studies using CEA increased by 23 percentage points while the proportion of studies using CBA decreased from 31% to 12% between 1980–84 and 1995–99. CUA remained the least common analytical technique used over all time intervals.

A cross tabulation of intervention type by analytical technique indicated that CEA was the most common technique for all intervention types. CBA tended to be used to evaluate health prevention and detection/diagnostic interventions, while CUA was usually applied to evaluations of health prevention interventions and health treatments.

Journal type

As economic evaluations typically represent multi-disciplinary research, the choice of publication venue is broad. Publication of economic evaluations by journal type is presented in table 4. Publication in journals of paediatrics/perinatal medicine was the most common venue for all time intervals and increased as a proportion of the total over time. Paediatric economic evaluations appearing in other sub-speciality journals also increased over time, while the proportion of publications in public health and general medicine journals decreased over time. There was also an increase in the proportion of publications of paediatric economic evaluations in journals devoted to health economics/health policy/methods.

A cross tabulation of journal type by analytical technique demonstrated that while CEAs tended to be concentrated in paediatrics/perinatal medicine journals, CBAs and CUAs were more evenly distributed across journals of paediatrics/perinatal medicine, sub-speciality medicine, public health and general medicine.

DISCUSSION

Just as the rate of adult economic evaluations continues to grow rapidly,^15,16 the number of paediatric studies published every year is rising. This growth reflects the importance of economic evaluations to assess the economic benefit of interventions, programmes, and services for children in a healthcare climate characterised by economic constraints and difficult allocation decisions. Significant growth in the number of paediatric economic evaluations was observed over the 20 year study period, with the number of publications approximately doubling every 5 years.

By definition, economic evaluations represent a balance of costs against consequences, with health consequences expressed as a particular type of outcome measure. Outcome measures that examined the number of cases of a disease, a condition, or an abnormality were the most frequent during the study period and are consistent with the preponderance of health prevention interventions. These measures represent intermediate outcomes. More final outcomes would include mortality, life years gained, and QALYs, and are preferable for allocation decision making.¹⁸ While the use of final outcomes represented only 20% of the total, they did demonstrate a sustained growth in frequency over the study period. Final outcome measures are particularly problematic for the paediatric population, unless the study is focused on a terminal disease of childhood such as a severe musculoskeletal disorder or cancer. Currently no published health economic models exist for measuring life years gained over long time horizons that include periods of maturation, development, and rapid physiological change.

A challenge in building the PEDE database was assigning an analytical technique. Given the high frequency of evaluations of preventive measures for infectious disease, many of the studies were self labelled as CBAs. In addition, the early publications pre-dated the widespread acceptance of the conventional analytical techniques defined as CEA, CBA, CUA, and CMA. These early studies tended to be self labelled as CBAs regardless of the actual analytical technique used. To circumvent the problem of mislabelled studies, the research team ignored the authors’ label and assigned one based on a careful reading of the methods used. Thus, many so-called CBAs that examined the incremental cost of cases prevented were relabelled as CEAs. However, if a so-called CBA monetarised the outcome measure (even if it did so poorly), the study remained a CBA. Our results indicated that true CEAs accounted for a large majority of paediatric health economic evaluations, a finding similar to the adult literature.¹⁷ CUAs were rare, even in 1995–99. This is probably due to the difficulty in ascertaining life years gained, as discussed above, as well as the challenge of estimating utilities for health states in children. As paediatric economic evaluation becomes more prevalent, a deeper understanding of the methodology should improve the author’s specification of the analytical technique. Otherwise, persistent misuse of the terms CBA and CEA will continue to hamper the conduct of literature searches for the purpose of methodological research in this field.¹⁸

Examining the venue chosen for journal publication can reveal the motivation behind the research, for example to influence clinical decision-making, to advance the methodology in the field, or to influence policy. The increasing rate of publication in paediatric and sub-speciality journals suggests that there is a growing appreciation of the importance of economic evaluation in paediatric medicine among clinical specialists. However, publication in a medical journal alone may be inadequate. As an applied form of research, the results of health economic evaluation must be disseminated in a manner that will best aid policy decision making. While publication in a highly regarded international speciality journal may be desirable from a clinician’s or academician’s perspective, it may fail to reach the responsible decision makers in regional governments or health organisations unless accompanied by a targeted dissemination strategy.

A particular challenge in identifying paediatric economic evaluations of healthcare interventions relates to the way ‘health’ is defined. In the paediatric population, health is intricately connected to behaviour and development. Many health problems in children are manifested as behavioural changes for which social services and educational interventions are advocated. Multiple care settings are involved in the delivery of care to children. These include physicians’ offices, clinics, schools, homes, and community agencies. Our focus on scanning only the health literature resulted in the exclusion of economic evaluations of psychological or social service interventions that may be relevant for child health. A greater understanding of child health that integrates psychological, educational, and social service interventions will necessitate a broader approach to assessing health outcomes in children.

In the expanding field of health economics, many researchers focus on methods research. The PEDE database serves as a valuable tool for exploration of methodological challenges in the conduct of paediatric health economic evaluation. In addition, researchers interested in preparing a systematic review and/or meta-analysis of a particular paediatric disorder or aspect of economic evaluation will derive great benefit from such a resource. Plans are underway to update the PEDE database on an annual basis. A web based, user friendly searchable database with appropriate links to paediatric health services research groups, health economic associations and information synthesis organisations is available at http://pede.bioinfo.sickkids.on.ca/pede.