Figures
Abstract
Background
Integrated Management of Childhood Illness (IMCI) is a strategy to reduce mortality and morbidity in children under 5 years by improving case management of common and serious illnesses at primary health care level, and was adopted in South Africa in 1997. We report an evaluation of IMCI implementation in two provinces of South Africa.
Methodology/Principal Findings
Seventy-seven IMCI trained health workers were randomly selected and observed in 74 health facilities; 1357 consultations were observed between May 2006 and January 2007. Each health worker was observed for up to 20 consultations with sick children presenting consecutively to the facility, each child was then reassessed by an IMCI expert to determine the correct findings. Observed health workers had been trained in IMCI for an average of 32.2 months, and were observed for a mean of 17.7 consultations; 50/77(65%) HW's had received a follow up visit after training. In most cases health workers used IMCI to assess presenting symptoms but did not implement IMCI comprehensively. All but one health worker referred to IMCI guidelines during the period of observation. 9(12%) observed health workers checked general danger signs in every child, and 14(18%) assessed all the main symptoms in every child. 51/109(46.8%) children with severe classifications were correctly identified. Nutritional status was not classified in 567/1357(47.5%) children.
Conclusion/Significance
Health workers are implementing IMCI, but assessments were frequently incomplete, and children requiring urgent referral were missed. If coverage of key child survival interventions is to be improved, interventions are required to ensure competency in identifying specific signs and to encourage comprehensive assessments of children by IMCI practitioners. The role of supervision in maintaining health worker skills needs further investigation.
Citation: Horwood C, Vermaak K, Rollins N, Haskins L, Nkosi P, Qazi S (2009) An Evaluation of the Quality of IMCI Assessments among IMCI Trained Health Workers in South Africa. PLoS ONE 4(6): e5937. https://doi.org/10.1371/journal.pone.0005937
Editor: Jacqueline Ho, JARING, Malaysia
Received: January 31, 2009; Accepted: April 28, 2009; Published: June 17, 2009
Copyright: © 2009 Horwood et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: All funding was provided by Department of Child and Adolescent Health and Development of the World Health Organization, Geneva. http://www.who.int/child_adolescent_health/ Dr. Qazi from WHO provided support throughout the process of designing the study, analysing the results and writing the paper.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In developing countries 9.7 million children under five years of age die every year [1], most deaths are from preventable and easily treatable diseases [2], and in a small number of developing countries [3]. It is estimated that over 60% of global child deaths could be prevented by available and affordable interventions [4], and their effective delivery is critical for achieving the Millennium Development Goal for child survival [5]. Integrated Management of Childhood Illness (IMCI) is a child survival strategy developed by the World Health Organisation (WHO) and United Nations Children's Fund (UNICEF) [6]. IMCI aims to improve coverage of essential child health interventions by improving case management skills of first level health workers, strengthening the health system for effective management of sick children, and promoting good community child care practices [7]. South Africa adopted IMCI as the standard of care for children in 1997, and is one of 43 African countries to do so [8].
IMCI case management training equips health workers with skills to manage children for a combination of illnesses, identify those requiring urgent referral, administer appropriate treatments, and provide relevant information to child carers. WHO recommends that newly trained IMCI practitioners receive follow-up visits from IMCI supervisors, starting 4–6 weeks after training, to assist them in transferring their newly acquired skills to the workplace [9]. IMCI implementation has been shown to improve the quality of management of sick children [10], [11], [12], and IMCI trained health workers communicate better with care-givers [13].
However, previous reviews have not described healthworkers' assessments of children in detail, and have used the observed child as the unit of analysis. In this article we report the results of an evaluation of the performance of IMCI trained health workers, conducted in two provinces in South Africa. We undertook a large number of observations overall, and for each health worker, so we are able to use the health worker as the unit of analysis and describe in detail how health workers assess and classify sick children. This provides a comprehensive picture of IMCI implementation in routine clinical practice, from which we are able to identify gaps in implementation, and suggest solutions.
Methods
Study site and population
IMCI guidelines in South Africa were adapted to include a component for management of HIV infected children, and evaluation of this component was a primary objective of this study. IMCI trained health workers (HW's) working in first level health facilities in Limpopo and Kwazulu-Natal (KZN) provinces, South Africa, were randomly selected for inclusion in the study. Health workers without IMCI training were excluded. IMCI implementation began in 1998, and at the time of our study 1325 health workers had been trained in Limpopo Province and 1300 in KZN, comprising 47% and 32% of health workers seeing sick children in PHC clinics respectively.
Training and Data Collection
Two IMCI experts visited facilities to collect data in each province. They were trained in study methods for two weeks by the investigators (CH, KV); data collection tools and methodology were piloted in two health facilities. All IMCI experts had previously attended the 11 day IMCI training course and the IMCI facilitators' course, and were experienced IMCI course directors.
The consultation by the health worker was observed by one IMCI expert who recorded the activities and findings without intervening. Activities recorded included whether the health worker referred to the IMCI chart booklet during the consultation, as would be expected if IMCI was being implemented correctly. During an IMCI consultation, health workers assess first for general danger signs, they then assess the four main symptoms (cough or difficult breathing, diarrhoea, fever, ear pain) and nutritional status. A classification is then made for each main symptom present, according to the signs identified during the assessment of the child. Thereafter, the second IMCI expert assessed the same child independently, and these findings were considered to be the gold standard for analysis of health worker performance. If the management of the child was incorrect, this was changed by the second IMCI expert as appropriate. Each health worker was observed for 20 consultations with sick children aged 2–59 months presenting consecutively to the health facility, or for 3 days if 20 observations had not been completed in that time.
The IMCI experts used standardized data collection instruments to record data about health workers' previous training and supervision; assessments by the health worker and the IMCI expert; and resources available at the clinic to support IMCI implementation. To monitor quality during data collection, the principal investigator visited the teams at least monthly, and all completed forms were checked for quality and completeness.
Consent and ethical approval
Written informed consent was obtained from carers of children for observation of the consultation with the health worker, and for the second assessment of the child by the IMCI expert. Health workers and observed children were allocated codes and no identifying information was recorded.
The study was conducted in partnership with the South African Department of Health (DOH), and all first level health workers in the two provinces were informed by the DOH that a survey of child health practices was to be undertaken. Participants were not told ahead of time that they had been selected, or that IMCI in particular was being evaluated, and because they were employed by the DOH they were required to participate.
Ethical approval was obtained from the Biomedical Research Ethics Committees of the University of KwaZulu-Natal Medical School, Durban, and WHO, Geneva.
Sample Size
A major objective of this study was to assess implementation of the IMCI HIV component. The sample size calculation was based on the assumption that 80% (+/−10%) of health workers would correctly classify for HIV in all 20 cases they assessed, compared to the IMCI expert. The sample was calculated as 62 health workers. IMCI trained health workers were randomly selected from a list of all IMCI trained health workers in each province using computer generated random numbers [14]. Sampling was stratified by province with equal numbers of health workers selected from each province.
However, an interim analysis found that only 26% (+/−10%) of health workers had correctly classified all children, the sample was therefore recalculated and increased to 77 health workers. The results of the HIV implementation assessment are reported elsewhere.
Data management and analysis
Pre-coded data were double entered, cleaned and validated using Epi-info (version 6.04). Analysis was conducted using SPSS (version 13.0) and Stata (version 9). The proportion of health workers who referred to the chart booklet and how frequently, was used as an indicator of whether observed health workers were implementing IMCI. To determine the performance of observed health workers, their assessments were compared to those made by the IMCI experts, which were considered to be ‘correct’ for purposes of analysis. To assess the performance of each health worker during the period of analysis, the proportion of observed children correctly assessed for each main symptom was calculated for each health worker. Using the child as the unit of analysis, we then calculated the proportion of children with each main symptom who were assessed correctly, assessed incorrectly or not assessed at all, by observed health workers. We then calculated the proportion of children with each IMCI classification that were correctly classified by observed health workers, using the child as the unit of analysis. 95% confidence intervals were calculated for all performance indicators.
Results
The consultations of 77 IMCI trained health workers working in 74 primary health care clinics in KZN and Limpopo provinces were observed between May 2006 and January 2007. Each health worker was observed for a mean of 2.7 days and 17.7 consultations.
Training of observed health workers
All observed health workers were registered nurses with a minimum of 3 years nursing training, and had attended an 11day IMCI training course, but most had no other special training in child health. The time since being trained in IMCI was an average of 32.2 months. Most health workers had received at least one follow up visit following IMCI training (table 1).
The average number of nurses on the staff establishment at clinics where we undertook our observations was 6, and on average 74% of these had been trained in IMCI. In 50/74 (67%) clinics visited, more than 60% of nurses were IMCI trained.
Performance of observed health workers during the observation period
31/77(40%) health workers referred to the IMCI chart booklet during every observed consultation, 35(45%) did so during some observed consultations, and only one health worker never referred to the chart booklet during the period of observation.
During the period of observation, 9 (12%) health workers asked about three general danger signs (unable to drink or breastfeed, vomiting everything, and convulsions with this illness) in every child, and 14 (18%) asked about all four main symptoms in every child. 7/9 (78%) health workers who checked the danger signs in every child also checked all main symptoms in every child. Only 17 (22%) health workers plotted the weight of all children. Depending on the presenting complaints of children presenting to the facility, each observed health worker assessed children with different symptoms and signs. Table 2 shows the performance of each health worker in classifying the children assessed during the observation period.
No association was found between health worker performance and whether the health worker had received a follow up visit by a supervisor, or the time since being trained in IMCI (data not shown). However the number of health workers in the sample was insufficient to exclude such an association.
Classification of observed children by health workers
During the 1357 observed consultations, health workers asked about three general danger signs in 795(58.6% CI: 49.8%–66.9%), and the four main symptoms in 815(60.1% CI: 51.0%–68.5%) children. Health workers did not ask about cough in 123(9.1% CI: 6.3%–12.9%) children, diarrhoea in 297(21.9% CI: 15.7%–29.6%) children, fever in 310(22.8% CI: 17.1%–29.8%) children, and ear problems in 409 (30.1% CI: 23.0%–38.3%) children. The performance of health workers in classifying observed children for each main symptom is shown in table 3.
Of 112 children assessed as having a severe classification or a danger sign by the IMCI expert, 52 (46.4% CI: 35.5%–57.7%) were also given a severe classification by the health worker. Health workers' performance in identifying each IMCI classification is shown in table 4.
Health workers either did not assess, or did not classify, for malnutrition in 567/1357(41.8% CI: 34.2%–49.8%) children (table 3), but the weight was plotted correctly on the growth chart in 1060/1357(78.1% CI: 72.9%–82.5%) children. The findings were explained to the mother in only 624(58.9% CI: 52.7%–64.8%) cases.
Discussion
Our findings show that IMCI is being widely implemented in clinics in South Africa several years into the expansion phase. Most clinics visited had good coverage with IMCI trained health workers, and despite the average time since training being almost three years, all but one health worker used the IMCI guidelines during observed consultations. However the IMCI assessment was not applied consistently and comprehensively, and activities not related directly to the presenting complaint were frequently omitted.
More observations were done in this study, both in total and of each health worker, than previous IMCI evaluations, allowing us to describe health worker performance in more detail. Health workers' performed best in assessing cough and dehydration, but even with these symptoms, only a small proportion of health workers assessed more than 80% of children correctly. The most common reason for health workers' not classifying correctly was failure to ask about the symptom or to make a classification at all, rather than making an incorrect classification. Few health workers consistently asked about all main symptoms, particularly later in IMCI assessment, indicating that incomplete assessments rather than simply lack of skills often leads to poor IMCI implementation.
Health workers' performance in identifying different classifications shows that health workers frequently fail to identify children with moderate or severe classifications, and perform best at identifying common, mild illnesses where no specific treatment is required. Less than half of severely ill children who required urgent referral to hospital were identified by IMCI trained health workers. Correct assessment of moderate or severe classifications depends on health workers' ability to identify specific signs, whereas mild classifications are usually based on the absence of these signs. For example, when assessing a child with a cough, identification of fast breathing or chest indrawing leads to a classification of pneumonia or severe pneumonia, whereas failure to identify these signs would lead to the mild classification of no pneumonia: cough or cold. It may be a lack of skills in identifying those specific signs required to make severe classifications that leads to poor performance, so those children most at-risk do not receive appropriate treatment.
Nutritional assessments were also poorly implemented; many children were not assessed for nutrition, most children with malnutrition were not identified, and feeding advice was frequently not given where indicated. Interventions and advice about nutrition, particularly promotion of breastfeeding and counselling about complementary feeding, have been shown to substantially improve child mortality [15]. Thus, failure to implement this aspect of IMCI will have a major impact on the potential for the IMCI strategy to improve child survival. A review of training materials and methods related to nutrition and identification of children with severe illnesses, could improve performance in these important areas of practise.
IMCI has been shown to improve care of children at first level [10], [11], [12], but poor adherence to IMCI guidelines has been repeatedly described [12], [16], [17], [18]. If IMCI implementation is to achieve sufficient coverage to make a difference to child mortality, it is critical that strategies are developed to achieve and maintain high quality health worker performance. Our results suggest that strategies to encourage health workers to apply the IMCI assessment comprehensively, including the nutritional assessment, would lead to an improvement in health worker performance.
Our results also highlight the importance of health workers' achieving competency at identifying signs of severe disease during IMCI training. Previous evaluations have shown that health worker performance is adversely affected when the amount of clinical practice included in IMCI training is reduced [16], as may occur when training is decentralised. So interventions to improve health worker performance should include ways of ensuring that competency in identifying the severe signs used in the IMCI assessment is achieved and maintained. A formal assessment could be introduced for IMCI practitioners on completing the training, and regular updates for IMCI practitioners could ensure that these skills are maintained, as well as providing support for practitioners in the workplace. Other methods of improving implementation like awarding clinics ‘IMCI excellent’ accreditation could be used to motivate practitioners.
The strengths of this study are that we observed large numbers of health workers and for more consultations than previous evaluations of IMCI implementation, so that analysis could be done at the health worker level. All our IMCI experts were experienced IMCI facilitators, and able to provide a reliable gold standard. The influence of the observer's presence on health worker performance was minimised by the large number of observations conducted over several days, so subject bias was reduced by habituation. Limitations of the study include not evaluating health workers' ability to identify particular signs, or treatments given to children, and no measure was taken of inter-rater reliability. We were also unable to determine reasons for poor performance in sufficient detail, including any relationships that may have existed between health worker performance and IMCI supervision, or time since training.
Further research is required to investigate the factors leading to poor health worker performance, which is frequently ascribed just to a lack of knowledge and skills. Health workers often find it difficult to transfer new skills to the work place, and to maintain these skills, especially as IMCI consultations take longer [19]. Implementing and sustaining IMCI follow up after training has been shown to be difficult in several previous evaluations of IMCI [11], [12], [16], [20]. However, supervision has been shown to improve performance [11] and may also improve motivation and job satisfaction. The role of IMCI supervision in IMCI implementation and different models for provision of supervision should be investigated further.
In conclusion, IMCI can improve quality of care for sick children, and is being implemented in those countries where most child deaths occur. In our setting almost all IMCI trained health workers were using IMCI to assess children, but incomplete implementation means IMCI is failing to achieve maximum benefits for child survival. Improvements in training and supervision can go some way to addressing these problems, but further research is required to fully understand the determinants of health worker performance, both in the long and short term and strategies for maintaining IMCI skills over time should be evaluated. Effective solutions to the problem of scaling up IMCI, and other public health interventions, are needed to bridge the gaps between knowledge and practise, and to achieve universal coverage of critical interventions to improve child survival.
Acknowledgments
We are grateful for the support of the child and adolescent health sub-directorate of the South African department of health, in particular Ntombi Mazibuko. Thanks to Beatrice Mlati and Janet Dalton, maternal and child health program managers from Limpopo and KZN provinces for their very valuable help with logistics. Thanks also go to Steve Reid from the Centre for Rural Health, Ann Robertson and Steve Donohue from Limpopo Department of Health, and Ruth Bland from the Africa Centre for Health and Population studies.
We would like to thank the data collection teams who worked so hard, the staff of the participating clinics for their support, and all the mothers and infants who agreed to participate.
Shamim Qazi is a staff member of the World Health Organization. The expressed views and opinions do not necessarily express the policies of the World Health Organization.
Author Contributions
Conceived and designed the experiments: CMH KV SAQ. Performed the experiments: CMH KV LH PN. Analyzed the data: CMH KV NR LH PN SAQ. Wrote the paper: CMH KV NR LH PN SAQ.
References
- 1.
UNICEF (2007) State of the world's children 2008: child survival. New York: UNICEF.
- 2. Bryce J, Boschi-Pinto C, Shibuya K, Black RE (2005) WHO estimates of the causes of death in children. Lancet 365: 1147–1152.
- 3. Black RE, Morris SS, Bryce J (2003) Where and why are 10 million children dying every year? Lancet 361: 2226–2234.
- 4. Jones G, Steketee RW, Black RE, Bhutta ZA, Morris SS (2003) How many child deaths can we prevent this year? Lancet 362: 65–71.
- 5. Bryce J, Black RE, Walker N, Bhutta ZA, Lawn JE, et al. (2005) Can the world afford to save the lives of 6 million children each year? Lancet 365: 2193–2200.
- 6. Tulloch J (1999) Integrated approach to child health in developing countries. Lancet 354: Suppl 2SII16–SII20.
- 7. Bryce J, Victora CG, Habicht JP, Black RE, Scherpbier RW (2005) Programmatic pathways to child survival: results of a multi-country evaluation of Integrated Management of Childhood Illness. Health Policy Plan 20: Suppl 1i5–i17.
- 8.
WHO (2004) Child and adolescent health and development progress report 2002–2003. Geneva: pp. 63–66. ISBN 92 4 159223 0 ISBN 92 4 159223 0.
- 9.
WHO (1999) Guidelines for follow up after training. World Health Organization.
- 10. Amaral J, Gouws E, Bryce J, Leite AJ, Cunha AL, et al. (2004) Effect of Integrated Management of Childhood Illness (IMCI) on health worker performance in Northeast-Brazil. Cad Saude Publica 20: Suppl 2S209–219.
- 11. El Arifeen S, Blum LS, Hoque DM, Chowdhury EK, Khan R, et al. (2004) Integrated Management of Childhood Illness (IMCI) in Bangladesh: early findings from a cluster-randomised study. Lancet 364: 1595–1602.
- 12. Armstrong Schellenberg JR, Adam T, Mshinda H, Masanja H, Kabadi G, et al. (2004) Effectiveness and cost of facility-based Integrated Management of Childhood Illness (IMCI) in Tanzania. Lancet 364: 1583–1594.
- 13. Gouws E, Bryce J, Habicht JP, Amaral J, Pariyo G, et al. (2004) Improving antimicrobial use among health workers in first-level facilities: results from the multi-country evaluation of the Integrated Management of Childhood Illness strategy. BullWorld Health Organ 82: 509–515.
- 14.
(1998–2009) Random-org.
- 15. Bhutta ZA, Ahmed T, Black RE, Cousens S, Dewey K, et al. (2008) What works? Interventions for maternal and child undernutrition and survival. Lancet 371: 417–440.
- 16. Pariyo GW, Gouws E, Bryce J, Burnham G (2005) Improving facility-based care for sick children in Uganda: training is not enough. Health Policy Plan 20: Suppl 1i58–i68.
- 17. Arifeen SE, Bryce J, Gouws E, Baqui AH, Black RE, et al. (2005) Quality of care for under-fives in first-level health facilities in one district of Bangladesh. Bull World Health Organ 83: 260–267.
- 18. Rowe AK, Onikpo F, Lama M, Cokou F, Deming MS (2001) Management of childhood illness at health facilities in Benin: problems and their causes. Am J Public Health 91: 1625–1635.
- 19. Adam T, Manzi F, Schellenberg JA, Mgalula L, de Savigny D, et al. (2005) Does the Integrated Management of Childhood Illness cost more than routine care? Results from the United Republic of Tanzania. Bull World Health Organ 83: 369–377.
- 20. Huicho L, Davila M, Campos M, Drasbek C, Bryce J, et al. (2005) Scaling up integrated management of childhood illness to the national level: achievements and challenges in Peru. Health Policy Plan 20: 14–24.