Article Text
PDF
Abstract
Objectives Handheld echocardiography (HAND) has good sensitivity and specificity for rheumatic heart disease (RHD) when performed by cardiologists. However, physician shortages in RHD-endemic areas demand less-skilled users to make RHD screening practical. We examine nurse performance and interpretation of HAND using a simplified approach for RHD screening.
Methods Two nurses received training on HAND and a simplified screening approach. Consented students at two schools in Uganda were eligible for participation. A simplified approach (HAND performed and interpreted by a non-expert) was compared with the reference standard (standard portable echocardiography, performed and interpreted by experts according to the 2012 World Heart Federation guidelines). Reasons for false-positive and false-negative HAND studies were identified.
Results A total of 1002 children were consented, with 956 (11.1 years, 41.8% male) having complete data for review. Diagnoses included: 913 (95.5%) children were classified normal, 32 (3.3%) borderline RHD and 11 (1.2%) definite RHD. The simplified approach had a sensitivity of 74.4% (58.8% to 86.5%) and a specificity of 78.8% (76.0% to 81.4%) for any RHD (borderline and definite). Sensitivity improved to 90.9% (58.7% to 98.5%) for definite RHD. Identification and measurement of erroneous colour jets was the most common reason for false-positive studies (n=164/194), while missed mitral regurgitation and shorter regurgitant jet lengths with HAND were the most common reasons for false-negative studies (n=10/11).
Conclusions Non-expert-led HAND screening programmes offer a potential solution to financial and workforce barriers that limit widespread RHD screening. Nurses trained on HAND using a simplified approach had reasonable sensitivity and specificity for RHD screening. Information on reasons for false-negative and false-positive screening studies should be used to inform future training protocols, which could lead to improved screening performance.
Statistics from Altmetric.com
Introduction
Rheumatic heart disease (RHD) continues to cause significant morbidity and mortality in many low/middle-income nations. The 2010 Global Burden of Disease Study estimated that RHD accounted for 345 100 deaths in 2010.1 Secondary prophylaxis, in the form of monthly penicillin injections, is both a clinically and financially effective strategy for preventing disease advancement; however, early disease detection is critical as prophylaxis is best initiated prior to significant valvular damage.2–4 Echocardiographic screening has replaced auscultation as the most important diagnostic tool for RHD given its high sensitivity for early case detection.5–7 However, national echocardiographic RHD screening programmes remain almost non-existent due to the high start-up cost of echocardiography equipment and the lack of a highly specialised healthcare workforce needed to conduct echocardiographic screening.
Compared with standard portable echocardiography (STAND), handheld echocardiography (HAND) offers a lower price point and higher degree of portability. Recent data show that HAND performed by experienced physicians has excellent sensitivity and specificity for the detection of RHD.8 ,9 However, current diagnostic guidelines provided by the World Heart Federation (WHF) require continuous wave (CW) Doppler, which is not currently available with HAND.10 Recently, Lu et al11 developed a simplified approach for RHD screening with HAND, which showed high sensitivity for disease detection and reduced the need for STAND evaluation by 80%. Translating this approach to non-experts is essential if widespread adoption of echocardiographic screening is to occur in low-resource areas. The objective of this study was to evaluate the performance of this simplified screening approach in the hands of non-experts.
Methods
Study setting and population
We prospectively evaluated the performance of a simplified RHD echocardiographic screening protocol implemented by two non-experts (nurses). The ‘simplified approach’ used HAND equipment and real-time decision-making to determine if a subject was screen positive or screen negative. This approach was compared with the reference standard; cardiologists with expertise in identifying or diagnosing RHD using STAND and off-line interpretation of these images by cardiologists with expertise in RHD using the full 2012 WHF criteria.10
This study was conducted between June and August of 2014 in two public primary schools in Gulu, Uganda. The prevalence of RHD in this region has been shown to be quite high, with 4% of children showing evidence of latent RHD in previous studies.8 This region is served by a paediatric cardiologist stationed at the Gulu Regional Referral Hospital and a well-organised RHD clinic that ensures the availability of follow-up evaluation and secondary prophylaxis for identified cases of RHD.
All students attending these two schools were eligible for participation. Participants were assigned a unique research identification number, which was used to link HAND and STAND studies. Each participant underwent two echocardiograms in the same day: HAND performed by one of two non-experts, and STAND performed by a senior paediatric cardiology fellow (MP), both blinded to the other's findings. The headmaster of each school consented for school participation. Per local standards, parents of children <8 years of age provided informed consent, and children >8 years of age provided informed assent. Institutional review board approval was obtained from Makerere University, Kampala, Uganda, and Children's National Medical Center, Washington DC, USA.
HAND training
Two Ugandan nurses with 6 months’ experience in obtaining a limited echocardiography protocol for RHD using STAND received education on use of HAND equipment and on the simplified screening approach. These nurses were already competent in obtaining 2D and colour images in the standard parasternal long, parasternal short, and apical four-chamber views. However, they had no previous experience in identification of morphological or functional abnormalities of the left-sided valves. Training focused on basic left-sided cardiac anatomy, recognising mitral and aortic valve regurgitation, use of the HAND equipment and correct measurement of regurgitant jets using the built-in calliper on the HAND equipment. Training included approximately 4 h of physician-directed teaching, using a combination of computer-based training modules (WiRED International, http://www.wiredhealthresources.net/EchoProject/index.html), didactics and case studies. This was followed by a 2-day hands-on session with patients at the RHD clinic at the Gulu Regional Referral Center. Each nurse performed and interpreted a minimum of 50 studies using HAND over these 2 days with 1:1 or 2:1 supervision.
Echocardiographic protocol
Each student underwent HAND using the GE Vscan V.1.2 (GE Medical Systems, Milwaukee, Wisconsin, USA). The handheld echocardiographic equipment (Vscan) used a 17.1–3.4 MHz transducer. Two-dimensional and colour Doppler images are projected on a 8.9 cm display. Frame rates range from 25 to 30 Hz for black-and-white imaging and from 12 to 16 Hz for colour Doppler. The HAND protocol included 2D and colour Doppler in the parasternal long-axis and apical four-chamber and five-chamber views, with a total of 11–13 recorded images per examination. Parasternal short-axis views (originally included in the protocol by Lu et al) were not included. If mitral regurgitation (MR) or aortic regurgitation (AR) was noted, the frame was frozen and scrolled to the image showing the maximum colour jet. Measurement was undertaken with the calliper feature on the device and recorded. The simplified diagnostic criteria are listed in table 1. After the examination was complete, the nurses reported whether the screen was positive or negative. As per previously published recommendations, MR jet ≥1.5 cm measured from the vena contracta to end of the regurgitant jet in any view and/or the presence of any AR in any view was considered a ‘screen positive’ examination, and referral was made for secondary evaluation.11 An independent investigator directly observed 10% of the studies, and recorded the time of HAND acquisition and the total time of the encounter.
Simplified rheumatic heart disease diagnostic criteria
Students then underwent STAND evaluation using the GE Vivid Q (GE, Milwaukee, Wisconsin, USA) performed by a single physician (MP) who was blinded to the nurses’ examination and recommendations. Standard portable echocardiographic equipment used a 1.5–3.6 MHz transducer. Frame rates ranged from 25 to 35 Hz for 2D imaging and from 12 to 18 Hz for colour Doppler. The STAND protocol was the same as the HAND protocol, with the addition of the parasternal short axis and CW Doppler recordings of the mitral and aortic valves. These images were later downloaded onto an offline PACS system, and interpreted independently by two cardiologists with expertise in RHD using the 2012 WHF criteria to determine the reference standard diagnosis of normal, borderline RHD or definite RHD.10 Table 2 describes the 2012 WHF RHD diagnostic criteria. Any disagreement between the two reviewers was adjudicated by a third paediatric cardiologist. HAND studies were reviewed by a paediatric cardiologist if there was disagreement between the recommendation on HAND (screen positive or screen negative) and the final consensus diagnosis with STAND (normal, borderline RHD, definite RHD), and reasons for non-agreement categorised and recorded.
World Heart Federation criteria for diagnosis of RHD in patients ≤20 years old
Statistical analysis
Study data were collected and managed using the REDCap electronic data-capture system hosted at Children's National Medical Center.12 Participant demographic and echocardiographic characteristics are described using mean and (SD, or proportions where appropriate. Sensitivity and specificity, with 95% CIs, were calculated for appropriate identification of any disease (borderline or definite RHD) and for definite RHD alone. Additionally, the sensitivity and specificity of each non-expert was also independently calculated. All data were analysed using MedCalc for Windows V.12.2 (MedCalc Software, Ostend, Belgium).
Results
HAND studies were performed in 1002 primary students. Five STAND studies were not transferred to the PACS system due to technical difficulties, resulting in 997 students with paired studies. Of these, the audio recording of subject ID was missing or unintelligible in 41 HAND studies, and these were excluded from further analysis. This left 956 studies for comparison, 492 obtained by nurse A and 464 by nurse B. Student ages ranged between 5 and 17 years, with a mean age of 11.1 years (SD 2.5 years). There were 376 males (42.1%) and 580 females (57.9%). Basic echocardiographic information on these 956 students can be found in table 3. Using the reference standard, 913 (95.5%) children were classified as normal, 32 (3.3%) as borderline RHD and 11 (1.2%) as definite RHD. The majority (86%) of the children diagnosed with RHD had isolated mitral valve disease, with the remaining (14%) having mixed mitral and aortic valve involvement. No patients had isolated aortic valve disease. Furthermore, there were no cases of mitral or aortic stenosis in this population. The 41 excluded patients included 26 females (63.4%) with a mean age of 11.4 years, which was not statistically different from the included patients (p=0.55). The excluded patients included one borderline RHD with the remainder being normal.
Echocardiographic data (N=956)
Overall, the HAND protocol implemented by non-experts had a sensitivity of 74.4% (58.8% to 86.5%) and a specificity of 78.8% (76.0% to 81.4%) for detecting any (borderline or definite) RHD. Performance improved when limited to definite RHD with an overall sensitivity of 90.9% (58.7% to 98.5%). The sensitivity and specificity between the two non-experts are illustrated in table 4.
Sensitivity and specificity of the simplified approach
Table 5 compares the performance of the screening protocol with HAND to the final disease categorisation by STAND. There was non-agreement between 205 studies (21%). However, 95% of this non-agreement (194/205 studies) came from HAND ‘positive’ studies that on STAND evaluation were found to be normal. False positives were predominantly due to measurement of an erroneous colour jet that was not MR/AR, with a smaller number related to a lower cut-off for MR jet length than WHF guidelines (1.5 cm vs 2.0 cm), and incorrect measurement of regurgitant jets. Using a cut-off of 2.0 cm for MR jet length decreased sensitivity for RHD to 41.9%. Only 5% of the non-agreement (11/205 studies) came from HAND ‘negative’ studies that on STAND were determined to have latent RHD, with only one of these referral failures in a case of definite RHD. False-negative HAND studies were primarily due to detection of MR jet <1.5 cm or an MR jet, which was imaged, but not noticed in real time by the non-expert screener.
Performance of the simplified screening approach in comparison with the reference standard
On average, the length of encounter was 6.69 min (SD 2.54 min) with 4.97 min (SD 2.28 min) required to acquire the echocardiographic images. Nurse A had an average encounter time of 6.16 min (SD 1.19 min) with 4.57 min (SD 1.0 min) spent performing the HAND echocardiogram. Nurse B had an average encounter time of 7.84 min (SD 4.0 min) with 5.45 min (SD 3.18 min) dedicated to performing the HAND echocardiogram. Delays in HAND screening were primarily related to ‘freezing’ of the HAND device with overheating and short battery life.
Discussion
HAND is a promising technology for RHD screening, but few data exist on the performance of non-experts using HAND. This study demonstrated acceptable sensitivity (74.4%) and specificity (78.8%) for both borderline and definite RHD, with excellent sensitivity for definite RHD (90.9%). To our knowledge, this study is the first to evaluate reasons for false-positive and false-negative studies, which will inform future training and screening protocols.
We further abbreviated the simplified protocol developed by Lu et al,11 which was developed to optimise sensitivity and specificity for RHD screening using HAND in a similar population, by only obtaining images in the parasternal long and apical four-chamber and five-chamber views.8 From this prior experience, we felt that parasternal short-axis views were mainly useful for assessing mitral and aortic valve morphology. This protocol focuses on functional (MR/AR) rather than morphological criteria, which are more difficult to teach to non-experts, and have lower rates of inter-reviewer reliability even among experts.9 ,11 Despite using experts both for performance and off-line interpretation of HAND studies, the sensitivity and specificity (73.2% and 82.4%, respectively, with 97.9% sensitivity for definite RHD) were quite similar to the results of non-experts in the field, reported here.13
Although these results are encouraging, the number of false positives is a concern. In this study, 194/956 students (20%) with no disease were positive by HAND screen. When scaling RHD screening up to a national level, this could represent tens of thousands of false-positive referrals, which could overwhelm healthcare resources.
Different diagnostic criteria for MR between the simplified protocol and WHF 2012 criteria accounted for only 13% of observed disagreement. Our use of less stringent diagnostic criteria to improve sensitivity is not unique. In a pilot study of nurse-led RHD screening using STAND in Fiji, investigators chose a referral cut-off of 1 cm of MR or any AR, with lower specificity (41.9% to 76.7%).14 However, Mirabel et al,13 in the only other study examining non-physicians using HAND with simplified criteria, found that MR ≥2.0 cm or any AR had the best combination of sensitivity and specificity. In their field test, this simplified approach had sensitivity of 80.6% and specificity of 91.5% for RHD. This has not been replicated in Ugandan cohorts with HAND by either experts or non-experts (in the current study), with improved specificity at a cut-off of 2.0 cm, but unacceptably large decrease in sensitivity, suggesting differences in cohort rather than experience or training.11
However, more false positives were due to misidentification and measurement of erroneous colour jets than issues with criteria. Technological advances such as addition of CW Doppler to the HAND platform could improve specificity for pathological MR. However, the greater specificity reported by Mirabel et al13 using the same device likely reflects the significantly longer training protocol used by their investigators, which included 3 days of lecture followed by 30 h of hands-on sessions, followed by additional mentoring and reviewing of HAND images.13 As the nurses in our study were already competent in image acquisition, we employed a shorter training period for image interpretation, including both didactic and hands-on sessions. The difference in specificity between nurses in our study highlights the need for individualised evaluation and training schedules, with feedback. Further study is needed to optimise non-expert training and to share best practices.
False negatives were less common, with the vast majority in borderline RHD. In half of these cases, MR meeting referral criteria was recorded, but was missed in real time by our non-expert screeners, underscoring the need for further education. In the remaining studies, MR was correctly identified and measured, but jet length was <1.5 cm, which may be related to technological differences with HAND, resulting in a shorter jet length, which is an important consideration in choosing a cut-off point.9 ,11
Implementation of any large screening programme requires efficiency. The nurses were able to complete the HAND protocol in a timely manner, but were limited by technical issues similar to those previously reported, including ‘freezing’ of the HAND device with overheating and short battery life.8 Despite proactive steps, such as multiple devices per nurse and docking the device between usages, there were still multiple technical delays. As HAND technology evolves, such difficulties may decrease.
Limitations
This study has several limitations. Our school-based screening was conducted in a single community in Northern Uganda, where patterns of RHD may differ from other parts of the world. In particular, no cases of mitral stenosis were seen in our screening cohort, and the sensitivity and specificity of this protocol for finding these cases cannot be determined by this study. Second, in this study, we assessed the performance of only two non-experts, both nurses with previous training on ultrasound image acquisition. Further research is needed on training a variety of healthcare professionals and lay workers to determine the most cost and clinically effective workforce. Additionally, for large-scale expansion, standardised training protocols and determinations of competency must be developed.
Conclusion
Echocardiographic screening for RHD has enormous potential for global RHD control. A strategy of HAND by non-experts may overcome two of the most important barriers to echocardiographic screening: cost and lack of a highly skilled healthcare workforce. Sensitivity of nearly 90% for definite RHD and adequate specificity can be obtained by non-experts using a very short training period, although further refinement in training and evaluation protocols is needed to limit false positives and false negatives. While use of HAND by non-experts cannot yet be recommended as a public heath strategy, these data add to a growing body of literature that may ultimately make echocardiographic screening for RHD an affordable and sustainable reality in low-resource settings.
Key messages
What is already known on this subject?
Echocardiographic screening for rheumatic heart disease (RHD) has the potential to drastically reduce the global burden of RHD, the most common cardiovascular disease in children and young adults. Widespread implementation has been hampered by severe financial and workforce limitations in low-resource, RHD endemic areas.
What might this study add?
Non-experts, implementing a simplified protocol for image acquisition and interpretation with handheld echocardiography (HAND) had a sensitivity of 74.4% (58.8% to 86.5%) and a specificity of 78.8% (76.0% to 81.4%) for the detection of any (borderline or definite) RHD. Sensitivity improved to 90.9% (58.7% to 98.5%) for definite RHD. The most common reason for false-positive studies was identification and measurement of erroneous colour jets (n=164/194); the most common reason for false-negative studies was missed mitral regurgitation and shorter regurgitant lengths with HAND (n=10/11).
How might this impact on clinical practice?
Use of HAND by non-experts offers a potentially sustainable approach to reduce the global burden of RHD. Future training programmes for non-experts should focus on the development of a structured, replicable educational curriculum and standardised competency testing.
Acknowledgments
We would like to acknowledge the hard work and dedication of our RHD study nurses, Agnes Atim and Stella Agutia, and the school children, teachers and parents of Gulu, Uganda, for their participation and dedication to the completion of this study.
References
Footnotes
Contributors AB, CS, TA and MP were responsible for the study planning and data acquisition. Data analysis was performed by AB and MP. All authors were involved in the drafting and critical revision of the submitted manuscript.
Funding This work was supported by the National Institutes of Health and National Center for Advancing Translational Sciences (grant numbers UL1TR000075 and KL2TR000076). This work was also completed through the support of General Electric, who provided the echocardiography equipment used for study completion.
Disclaimer The content of this work is the responsibility of the authors, and does not necessarily represent the views of the National Center for Advancing Translational Sciences or the National Institutes of Health.
Competing interests None declared.
Ethics approval Children's National Medical Center Institutional Review Board, Makerere University Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Editorial
- Heart failure and cardiomyopathies
- Heartbeat