Objective Integrated Management of Childhood Illness (IMCI) is the leading clinical protocol designed to decrease under-five mortality globally. However, impact is threatened by gaps in IMCI quality of care (QOC). In 2010, Partners In Health and the Rwanda Ministry of Health implemented a nurse mentorship intervention Mentoring and Enhanced Supervision at Health Centres (MESH) in two rural districts. This study measures change in QOC following the addition of MESH to didactic training.
Design Prepost intervention study of change in QOC after 12 months of MESH support measured by case observation using a standardised checklist. Study sample was children age 2 months to 5 years presenting on the days of data collection (292 baseline, 413 endpoint).
Setting 21 rural health centres in Rwanda.
Outcomes Primary outcome was a validated index of key IMCI assessments. Secondary outcomes included assessment, classification and treatment indicators, and QOC variability across providers. A mixed-effects regression model of the index was created.
Results In multivariate analyses, the index significantly improved in southern Kayonza (β-coefficient 0.17, 95% CI 0.12 to 0.22) and Kirehe (β-coefficient 0.29, 95% CI 0.23 to 0.34) districts. Children seen by IMCI-trained nurses increased from 83.2% to 100% (p<0.001) and use of IMCI case recording forms improved from 65.9% to 97.1% (p<0.001). Correct classification improved (56.0% to 91.5%, p<0.001), as did correct treatment (78.3% to 98.2%, p<0.001). Variability in QOC decreased (intracluster correlation coefficient 0.613–0.346).
Conclusions MESH was associated with significant improvements in all domains of IMCI quality. MESH could be an innovative strategy to improve IMCI implementation in resource-limited settings working to decrease under-five mortality.
- General Paediatrics
- Health services research
- Nursing Care
Statistics from Altmetric.com
Integrated Management of Childhood Illness (IMCI) was introduced to address the major causes of global mortality in children under age five.1 IMCI is an algorithmic symptomatic approach to paediatric visits designed to improve case management, health systems support, and family and community practices.2
Over 100 countries use IMCI, and studies have shown that IMCI improves healthcare quality, health worker performance, and can result in decreased under-five mortality and health cost savings.3–5 However, many countries have experienced significant barriers to widespread implementation, including poor training coverage, inadequate equipment and infrastructure, and political and financial constraints. Even in areas where IMCI has been implemented, optimal clinical benefits were not observed.6–10 One of the main challenges to achieving and maintaining the benefits of IMCI has been the quality of the IMCI-related services provided.
Supportive supervision following training has improved health worker performance in service delivery, including IMCI, in a number of settings.11–15 For example, healthcare workers in Uganda who received at least one supervisory visit including clinical observation following their IMCI training performed significantly better than those without this supervision.11 However, effective supervision models have been difficult to maintain.12–14
The Rwanda Ministry of Health (MOH) began the adaptation, training, and implementation of the IMCI protocol and case recording forms (CRF) in 2006 (see online supplementary file), with national scale-up of nurse training by 2008.16 However, similar to the findings of global evaluations,3 2 years after national training, significant barriers to IMCI implementation remained.16 These included IMCI use only on intermittently scheduled days, shortages in IMCI-trained nurses due to staff attrition, mismatch of nurse training with clinical assignment and limited on-site supervision. Despite remarkable progress, Rwanda still has significant rates of under-five mortality (76 per 1000 live births) and infant mortality (50 per 1000 live births), making improving IMCI a national priority.17
Partners In Health (PIH) has collaborated with the Rwanda MOH since 2005 to strengthen the healthcare delivery system in selected remote and underserved districts in the country. PIH is a global non-profit organisation committed to fighting disease and poverty in resource-limited settings together with local partners. We describe the impact of the addition of clinical and quality improvement (QI) mentorship to standard IMCI training on IMCI quality of care in PIH-supported districts.
This intervention and study occurred in all 21 nurse-led health centres (HCs) in two rural districts in Rwanda (southern Kayonza and Kirehe), serving an estimated population of 530 000.18
The Mentoring and Enhanced Supervision at Health centres (MESH) programme was designed by PIH and the Rwanda MOH to build upon the foundation of standard IMCI training with: (1) regular visits by a clinical mentor for supportive supervision and (2) support for QI efforts to address systems gaps that affect IMCI implementation (figure 1).19 Between December 2010 and May 2011, after formal IMCI training had been facilitated to meet the Rwanda MOH minimum standard of 60% coverage of nurses per HC, MESH was sequentially introduced to groups of four HCs to achieve full coverage of all facilities in two districts. One clinical mentor per district trained in IMCI, mentorship and QI visited each HC every 4–6 weeks. Mentor visits included clinical coaching in the use of IMCI tools (algorithm and CRFs) through case observation, case-based and didactic teaching, feedback of performance data and facilitation of QI activities.19 No other IMCI-strengthening interventions occurred during the study period.
The evaluation was a prepost intervention design. Data collection was performed after IMCI training and before MESH (baseline) and after 12 months of MESH (endpoint) at all 21 HCs by IMCI nurse expert observers using a case observation checklist. All sick children aged 2 months to 5 years presenting to the HC for the first time during an acute illness episode on the days of data collection were eligible for inclusion. Observations were excluded if the child was severely ill and the observer's immediate intervention was needed for appropriate patient care. For other patients, incorrect treatment prescribed by the nurses was recorded and then corrected by the observer prior to patient departure.
The case observation checklist was developed based on existing MOH and WHO tools20 and adapted to the Rwandan IMCI protocols. Other data collection tools included a mentor activity log and a quarterly HC survey to measure presence of essential IMCI-related equipment and medications.
Nurses provided written informed consent to observation, and the observers explained the purpose of the study to caregivers, who could opt-out. No patient identifiers were collected. The study was approved by the Institutional Review Board of Brigham and Women's Hospital and the Rwanda National Ethics Committee.
Sample size and data collection
Sample size was calculated to detect a 15% change in the primary outcome at the district level (ß=0.8) with an upward adjustment for potential clustering.21 ,22 The final target was calculated as 200 clinical observations per district per time point to account for district-level differences in population. Based on this calculation, data collection continued to reach a minimum target of 15 (Kirehe) and 20 (southern Kayonza) observations per HC over up to 3 days. Time limits were placed due to time and resource constraints. The median number of observations included at baseline was 10 (range 1–24) and 17 (range 2–28) and at endpoint was 15 (range 14–19) and 25 (range 25–30) for Kirehe and southern Kayonza, respectively. Data collection took a total of 6 months at baseline and 5 months at endpoint, with HCs averaging 12.6 months of intervention exposure. The final study sample included case observation of 292 consultations at baseline and 413 at endpoint.
Health worker adherence to IMCI assessment, classification, counselling and treatment was assessed through indicators and summary indices extracted from instruments based on those validated in the IMCI multi-country evaluation (MCE).23 Our primary outcome was the IMCI Integrated Assessment Index.23 The Integrated Assessment Index was calculated for each consultation as the mean of 13 equally weighted assessment tasks completed in >2-year-olds and 16 tasks in <2-year-olds and normalised on a scale from 0 to 1 (figure 2). If a task was missing from data collection, the index was calculated using the number of items for which data were recorded. Approximately 5% of checklist items were missing from data collection at baseline, which were excluded from the index denominator. Sensitivity analysis counting missing items at baseline as completed found no change in results.
Secondary outcomes included additional assessment, classification and treatment indicators, use of IMCI protocols, and variability in quality of care delivered across nurses.
Differences were analysed using t tests for continuous variables and χ2 tests for categorical variables. Population characteristics and the trend to improvement in Integrated Assessment Index were similar in each district in bivariate analyses; therefore, we present the results combined across districts. Most (96.7% (59/61)) nurses contributed multiple observations to the study sample. We measured the proportion of total variance in the index accounted for by the specific nurse provider at baseline and endpoint using the intracluster correlation coefficient (ICC). ICC was calculated using the ANOVA framework for the index outcome with nurse as the clustering factor.
Change in Integrated Assessment Index after controlling for potentially confounding factors was measured using mixed-effects regression. Because all nurses observed at endpoint were formally trained in IMCI, we could not separate improved training coverage from the impact of MESH in the multivariate analyses. Therefore, only nurses who were trained in IMCI were included in the multivariate analysis. Nurse was included as a random effect to account for clustering among consultations performed by the same nurse. Potential confounders were identified based on a conceptual framework derived from the MCE and included in the model a priori.24 Additional variables were considered for the model through backward selection with threshold p<0.05. Because MESH was implemented at the district level, an interaction term between district and preintervention or postintervention was included.
During the 12-month intervention period between baseline and endpoint, HCs received an average of 11.8 mentoring visits. Characteristics of our study population are described in table 1. No significant difference in HC or patient characteristics was seen between baseline and endpoint.
IMCI assessment, classification and treatment
Significant improvement was seen in almost all IMCI quality of care indicators. The primary outcome—the IMCI Integrated Assessment Index—improved significantly in both districts (figure 3). The index improved from 0.64 (SD 0.25) to 0.96 (SD 0.05) in children over 2 years of age, and from 0.61 (SD 0.23) to 0.92 (SD 0.09) among those under 2 years of age (p<0.001). All other quality of care measures for patient assessment improved significantly except for ‘weight checked against a growth chart’ category (table 2). Correct classification also improved significantly for all classifications (56.0% to 91.5%, p<0.001), and for pneumonia, diarrhoea and fever (58.7% to 98.7%, p<0.001).
Improvement in percentage of children given correct treatment per IMCI was also seen (78.3% to 98.2%, p<0.001). In subanalyses of specific clinical diagnoses, treatment improved in all disease categories measured; however, statistically significant improvement was only seen for pneumonia (71.4% to 98.6%, p<0.001). Counselling indicators dramatically improved from minimal activity at baseline to nearly all caregivers being counselled on appropriate hydration and reasons to return to the HC for care.
IMCI coverage and quality variability
Use of the IMCI CRF guides clinical decision-making per protocol. CRF use and nurse IMCI training status were used as measures of IMCI coverage among observed cases. The proportion of children seen using an IMCI CRF increased from 65.5% to 97.1% (p<0.001), with similar increase in treatment by an IMCI-trained nurse (83.2% to 100%, p<0.001). The variability in quality explained by the nurse performing the consultation decreased from baseline to endpoint (ICC 0.613–0.346).
The effect of MESH on mean IMCI Integrated Assessment Index was modified by district; therefore, stratified results are presented. Variables included in the final model were: district, age under 2 years, sex, presence of cough, presence of fever, HC duration of PIH support, proportion of essential oral and injectable drugs present at the HC at baseline, patient volume, and level of mentoring support during the intervention period. After restricting to IMCI-trained nurses and adjustment for potential confounding and clustering at the nurse level, MESH remained associated with a significant improvement in the Integrated Assessment Index in both districts: β-coefficient 0.29 (95% CI 0.23 to 0.34) in Kirehe and 0.17 (95% CI 0.12 to 0.22) in southern Kayonza (table 3). Other variables significantly associated with higher scores were child age <2 years (95% CI 0.01 to 0.05) and fever as a presentation (95% CI −0.06 to −0.02).
Effective delivery of IMCI can improve management of the main causes of under-five mortality globally; however, high quality IMCI care has been difficult to achieve in many resource-limited settings. We report the improvement in quality of IMCI care delivery through a nurse mentoring and QI intervention integrated into the existing health system. This finding remained significant after adjusting for a number of confounders and limiting to IMCI-trained nurses. While younger patient age was associated with a slightly lower index, potentially related to the higher task-load required for children under 2 years of age, improvement was also seen in both age groups.
Formal training has been the traditional approach to IMCI introduction, but is often followed by limited supervision. Training is typically expensive, and continuous staff turnover decreases the effectiveness and increases the cost. Given that MESH led to significant improvement in IMCI assessment among IMCI-trained nurses, MESH could be an important complement to amplify and sustain the impact of initial training on quality of care delivered.
While our sample characteristics and primary outcome did not differ by district in bivariate analyses, the effect of MESH was modified by the implementation district in the multivariate model. This interaction was likely due to the lower baseline index scores in Kirehe, allowing for a steeper improvement during the year of intervention than in southern Kayonza. However, both districts significantly improved after 1 year of MESH.
Our study also found that almost all other indicators of IMCI assessment, classification and treatment improved significantly after 1 year of MESH. One exception was the use of a growth chart for nutrition screening. We found this to be a particularly challenging area for nurses and mentorship in this area is ongoing.
Although no new trainings were conducted during the intervention period, we found improvement in IMCI coverage with 100% of children seen by an IMCI-trained nurse at endpoint. Ensuring management by a formally trained nurse through alignment of nurse assignments with training was targeted by MESH-supported QI work. Additionally, the increased CRF use supported through mentorship suggests improved use of IMCI protocols during under-five visits, as the forms are designed to guide protocol use.
Variability in care delivery between clinics and within providers has been used as a measure of poor quality.25 ,26 At baseline, 61.3% of variability in the index was explained by the nurse performing the consultation. This decreased after 1 year of MESH, with only 34.6% of variability explained by the specific nurse, reflecting a general increase in consistency as well as quality across nurses providing IMCI care.
There were some limitations in our study. This implementation research was not a controlled efficacy study; rather, it was designed to understand the effectiveness of a planned intervention aimed at addressing key constraints that have historically hindered achieving the potential impact of IMCI on child health.27 We adjusted for cofactors to make the baseline and endpoint as comparable as possible, but residual confounding could remain in this prepost design. For example, the Rwanda IMCI protocols were updated between our baseline and endpoint, resulting in longer protocols perceived by nurses to be more challenging. However, this would have biased our results towards the null. We also had a lower number of baseline observations than planned, but were still able to detect change in IMCI assessment in bivariate and multivariate analyses at the district level.
Another potential limitation is the Hawthorne effect, which would cause nurses to improve performance solely as a result of being observed. However, this bias would have been present at both time points and likely reduce detectable change. Last, we would have liked to formally assess the relationship between the MESH programme and traditional IMCI training. However, all cases observed at endpoint were managed by an IMCI-trained nurse; therefore, this analysis could not be performed.
In conclusion, we found that mentorship and systems-focused QI integrated into the routine supervision system is a feasible and effective strategy to bridge the IMCI implementation gap. This could be an important strategy to achieving the predicted impact from IMCI on child survival in similar settings, as well as improving quality of care in other protocol-driven areas including maternal health and HIV management. Further investigation is needed to explore the long-term effects of the MESH strategy on IMCI-trained staff and the cost-effectiveness of this combined mentorship and QI approach in improving the processes and outcomes of IMCI. Given the challenges of staff turnover and cost of formal trainings, study of the potential role of MESH as an alternative to the traditional training approach is also needed.
We gratefully acknowledge the support of the Doris Duke Charitable Foundation's African Health Initiative. We would like to thank Jennifer Bryce, EdD, MEd, for her technical consultation with use of IMCI multi-country evaluation measures and tools; Bethany Hedt-Gauthier, PhD, for her analytic support; and Karen Finnegan, MPH, for her support of study design and data collection. Finally, we would like to thank the IMCI mentors who work tirelessly to support the care of children in their communities.
Review history and Supplementary material
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Contributors HM: study conception and design, acquisition of data, training of data collectors and supervision of data collection, analysis and interpretation of data, manuscript drafting and critical revisions, administrative and technical support and supervision. MA: study conception and design, acquisition of data, supervision of data collection, analysis and interpretation of data, critical revision of the manuscript for important intellectual content, administrative and technical support. FRC: study conception and design, acquisition of data, training of data collectors, interpretation of data, critical revision of the manuscript for important intellectual content, administrative and technical support. CM: study conception and design, analysis and interpretation of data, critical revision of the manuscript for important intellectual content, statistical analysis. FN and SN: interpretation of data, critical revision of the manuscript for important intellectual content, administrative support and supervision of intervention and research. PCD: study conception and design, analysis and interpretation of data, critical revision of the manuscript for important intellectual content, obtaining funding, administrative and technical support. FN: analysis and interpretation of data, critical revision of the manuscript for important intellectual content, administrative and technical support. LRH: study conception and design, analysis and interpretation of data, critical revision of the manuscript for important intellectual content, obtaining funding, administrative and technical support, study supervision.
Funding Doris Duke Charitable Foundation's African Health Initiative.
Competing interests None.
Ethics approval Institutional Review Board of Brigham and Women's Hospital and the Rwanda National Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.