Article Text
Abstract
Background Globally, 2.5 million neonates died in 2018, accounting for 46% of under-5 deaths. Multiparameter continuous physiological monitoring (MCPM) of neonates allows for early detection and treatment of life-threatening health problems. However, neonatal monitoring technology is largely unavailable in low-resource settings.
Methods In four evaluation rounds, we prospectively compared the accuracy of the EarlySense under-mattress device to the Masimo Rad-97 pulse CO-oximeter with capnography reference device for heart rate (HR) and respiratory rate (RR) measurements in neonates in Kenya. EarlySense algorithm optimisations were made between evaluation rounds. In each evaluation round, we compared 200 randomly selected epochs of data using Bland-Altman plots and generated Clarke error grids with zones of 20% to aid in clinical interpretation.
Results Between 9 July 2019 and 8 January 2020, we collected 280 hours of MCPM data from 76 enrolled neonates. At the final evaluation round, the EarlySense MCPM device demonstrated a bias of −0.8 beats/minute for HR and 1.6 breaths/minute for RR, and normalised spread between the 95% upper and lower limits of agreement of 6.2% for HR and 27.3% for RR. Agreement between the two MCPM devices met the a priori–defined threshold of 30%. The Clarke error grids showed that all observations for HR and 197/200 for RR were within a 20% difference.
Conclusion Our research indicates that there is acceptable agreement between the EarlySense and Masimo MCPM devices in the context of large within-subject variability; however, further studies establishing cost-effectiveness and clinical effectiveness are needed before large-scale implementation of the EarlySense MCPM device in neonates.
Trial registration number NCT03920761.
- technology
- intensive care units
- neonatal
- neonatology
Data availability statement
Data are available in a public, open access repository. Data will be made available on completion of the secondary analyses of the data. We have ethics approval to deposit the data in an open access repository upon completion of the study.
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
What is already known on this topic?
Sustainable Development Goal 3’s focus on treatment and prevention of acute illness calls for development and optimisation of innovations for neonatal care in low-resource settings.
Continuous monitoring of neonatal heart and respiratory rates is rarely performed in low-resource settings due to poor availability of monitoring technologies and high acquisition costs.
Almost all studies of automated monitoring technologies have been conducted in high-resource settings, making it difficult to assess these technologies for use in low-resource settings.
What this study adds?
This is a novel evaluation of the accuracy of a contactless multiparameter continuous monitoring device compared with a reference standard in neonates in a low-resource setting.
We describe a method for and some of the challenges in performing monitoring device accuracy studies in neonates.
Our study indicates acceptable agreement between the EarlySense device and the Masimo Rad-97 reference device for measurement of heart and respiratory rates.
Introduction
High neonatal mortality rates persist in Sub-Saharan Africa, with 27 deaths/1000 live births, a rate 10 times higher than rates in high-resource settings.1 Interventions are urgently needed, particularly in low-resource African settings, if Sustainable Development Goal Target 3.2 of reducing neonatal mortality to 12 deaths/1000 live births is to be achieved by 2030.2
Major causes of neonatal mortality include complications of preterm birth, asphyxia, and infectious diseases which are preventable and treatable.3 Technologies that can detect vulnerability and prompt interventions have great potential to improve neonatal survival. Multiparameter continuous physiological monitoring (MCPM) devices could be key in identifying high-risk neonates. Studies have demonstrated that monitoring of vital sign trends can predict clinical deterioration and allow for timely intervention and improvement in health outcomes.4–8 However, neonatal MCPM devices are not available in most low-resource settings due to barriers such as the high cost of equipment and a lack of health personnel with appropriate training.9 10
A better understanding of if and how MCPM devices can be adapted and optimised for use in neonates in low-resource settings is critical. Our study’s primary outcomes were agreement of heart rate (HR) and respiratory rate (RR) measurements comparing the EarlySense (EarlySense, Israel) investigational device to the Masimo Rad-97 pulse CO-oximeter with capnography (Masimo Corporation, USA) reference device. We hypothesised that the EarlySense device would agree within a priori–defined thresholds for HR and RR measurements when compared with the reference device and show minimal bias.
Methods
Study design and procedures
We conducted an iterative prospective study at Aga Khan University, Nairobi (AKU-N), a tertiary healthcare facility in Kenya, to assess agreement of HR and RR measurements from the EarlySense investigational device with those from the Masimo Rad-97 reference device.10 The Masimo Rad-97 device was selected based on its capability for high-resolution data collection and neonatal capnometry and pulse oximetry. EarlySense’s contactless wireless piezoelectric sensor can be placed under a patient’s mattress to detect ballistic vibrations from respiratory chest wall movement and cardioballistic movements from ejection of blood (online supplemental appendix A).
Supplemental material
During Masimo Rad-97 reference device verification, we tested functionality and estimated within-neonate and between-neonate variability to pre-specify thresholds for agreement comparisons.11 We completed an initial round of open-label data collection from both devices in which complete reference device data were shared with EarlySense prior to analysis. Three rounds of closed-label analysis followed, in which the first 15–30 min of reference device data from each neonate were provided to EarlySense to confirm time synchronicity across data sources (online supplemental appendices B, C). All reference device data were provided to EarlySense on completion of each subsequent round of data analysis to optimise HR and RR detection algorithms.
Caregivers of neonates delivered at AKU-N during the study period were approached for recruitment into the study and trained study clinicians obtained informed consent prior to determining eligibility for enrolment (table 1). Enrolled neonates were simultaneously monitored by both devices (figure 1; online supplemental video 1).
Supplementary video
Data processing and selection
To ensure accurate, reliable and consistent data collection, we conducted the study in accordance with the Guideline for Good Clinical Practice/International Standards Organization (ISO) 14 155.
We randomly selected epochs of HR and RR with sufficient signal quality for each neonate recording (table 1). The first 5 min and last minute of recorded data, and first minute prior to and 5 min following any device or sensor disconnection were excluded to remove potentially poor-quality data typically present during sensor placement and removal. For the open-label round, we selected 20 epochs from ten different neonates. For each closed-label round, we selected ten epochs from 20 different neonates. No overlapping epochs were selected, and missing data were excluded from analyses. Median HR and RR values were calculated for the selected epochs (table 1). Manual RR counting was performed for each epoch using capnograms (online supplemental appendix D). Any RR epochs meeting exclusion criteria were replaced with another randomly selected epoch (table 1). See online supplemental appendix E for more details.
Statistical analysis
To quantify the agreement between the EarlySense and reference devices, the normalised bias (95% CI) and the normalised spread between the 95% limits of agreement (LOA) were calculated by dividing the bias and spread between the 95% LOA by the overall reference HR or RR mean value.12 Based on a verification phase, the acceptable a priori-defined normalised spread between the 95% upper and lower LOAs of 30% was selected for both RR and HR.11 We calculated the root-mean-square deviation (RMSD) for each comparison. Clarke error grids were constructed with zones of 20% discrepancy to improve clinical interpretability of results.
Results
We enrolled 94 neonates from 9 July 2019 to 8 January 2020, 76 of which were included for analysis (online supplemental appendix F). Three neonates withdrew prior to the minimum data collection time, and 15 neonates were excluded due to issues with data identification (incorrect date/times recorded) or time synchronisation (10), insufficient (less than 1 hour) length of recording (4) or poor data quality (1). There were six recordings where only one of the vital signs was usable for analysis due to poor data quality. There were no adverse events due to monitoring.
Equal numbers of female and male neonates from the neonatal high dependency unit (64.5%), postnatal and maternal wards (34.2%), and neonatal intensive care unit (1.3%) were included for analysis. Median gestational age was 38 (range 28–42) weeks and primary diagnoses were healthy post-delivery (35.5%), prematurity (27.6%), jaundice (17.1%), hypoglycaemic (6.6%), respiratory distress syndrome (5.3%), low birth weight (3.9%) and other (3.9%). We collected 280 hours of data (online supplemental appendix G), with a median recording length of 4 hours (range 73–423 min). See online supplemental appendix H for additional results.
HR analysis identified a small negative average bias (range −1.58 to −0.55 beats/minute) with normalised spread of LOA meeting the a priori–defined threshold for all rounds (table 2; figure 2). A marked decrease in normalised spread between 95% LOA between closed-label rounds 2 and 3 (20.1 vs 6.2%) was observed. All EarlySense HR measurements were within 20% of the reference device values (figure 3).
RR analyses showed a small positive average bias in the EarlySense device (range 0.17–1.62 breaths/minute) when compared with reference device RR manual counts and a small negative average bias (range −1.65 to −0.63 breaths/minute) when compared with RR median values (table 2; figure 4). Normalised spread between 95% LOA decreased between the open-label and first two closed-label rounds, with results from closed-label rounds 2 and 3 meeting the a priori–defined threshold for both RR manual counts and median values. A marked decrease in normalised spread between 95% LOA from closed-label rounds 1 and 2 (37.6 vs 24.8% for RR manual counts; 35.9 vs 19.2% for RR medians) was observed. Absolute and normalised spreads of 95% LOA for RR median values were consistently smaller than RR manual counts when comparing the EarlySense and reference devices. Of the data pairs from closed-label round 3, only three (1.5%) EarlySense RR values were outside 20% of the reference device values (figure 3).
Discussion
We found acceptable agreement between the EarlySense and reference devices for measurement of HR and RR in neonates by the final round of analysis, as evidenced by normalised spread between 95% LOA values below the a priori–defined threshold of 30%. The EarlySense algorithms were optimised between closed-label rounds 1 and 2 for RR, and between closed-label rounds 2 and 3 for HR. Upon optimisation, a marked narrowing of normalised spread between 95% LOA and reduction in bias was observed. Clarke error grids showed that all observations for HR and 197/200 for RR fell in region A (20% difference) (figure 3).
Selection of a reference standard is key when evaluating accuracy of a MCPM device. While ECG is the accepted reference standard for HR measurement, plethysmogram-derived HR, as used by the Masimo Rad-97 device, has been shown to be accurate even for comparisons of HR variability that require precise timing in identification of the heartbeat.13 The EarlySense device uses ballistocardiography (BCG). In comparison with ECG, heartbeats are less pronounced and more difficult to detect in BCG signals due to a lower signal-to-noise ratio14; yet, BCG signals have been shown to accurately estimate HR in adults.15
We selected capnography as the RR reference method.16 Agreement between RR measurements was impacted by within-neonate RR variability, which is particularly pronounced in neonates due to immature control of respiration, and normal and abnormal physiological responses such as periodic breathing and periods of apnoea.17 There are currently no recommended ISO standards for RR accuracy, though United Nations International Child’s Fund has suggested a maximum variability of ±2 breaths/minute in their acute respiratory infection diagnostic aid target product profile.18 This threshold may be unreasonably conservative when considering a neonate breathing at 60 breaths/minute, where a within-neonate variation of 2 breaths/minute would equate to a 3.3% variation. In setting the 30% threshold for the spread between 95% LOA, we considered the large within-neonate RR variability present in the verification phase data and avoided setting the target agreement threshold to be more restrictive than the within-neonate variation.11 We chose to normalise the spread between 95% LOA and used a percentage error threshold to allow for comparison of performance over a wide range of RR values. Use of a ±30% error threshold for LOA to determine whether to accept a new method in cardiac output method comparison studies was proposed in 1999 and has since been used extensively in the field.19
We compared EarlySense RR measurements to reference device RR manual counts and medians. Since clinical measurement of RR is typically determined by counting the number of breaths over a 1 min period, comparison of EarlySense RR values to RR manual counts was used as the primary comparison. However, manual count methods have several limitations. Capnograms can contain several different breathing patterns, such as periodic breathing and small-amplitude breaths, and it is unclear which breathing patterns should be included in a manual count and which should cause a minute to be excluded. In addition, rounding is a significant consequence of counting breaths. The rounding error may be as much as 6.7% if considering a RR of 15 breaths/minute and a rounding error of 1 breath/minute. We used a comparison between EarlySense RR and reference device RR median values since median values are more likely to reflect the underlying control of breathing. Medians are more robust than means, especially in the presence of extreme outliers, and are often used in automated device signal processing methods. This may explain the greater agreement when comparing the EarlySense RR to the reference device medians versus RR manual counts.
We reported RMSD values, as the accuracy of HR and RR has typically been reported as a percentage error rate (such as ±2%) compared with a reference standard.20 While RMSD is provided as a standardised metric when performing comparisons, RMSD does not consider the uncertainty in the reference method. In this evaluation, the normalised RMSD was 1.7% for HR, and 7.7% and 5.2% for RR manual counts and median values, respectively, in the final round (table 2). This was approximately 25% of the value of normalised spread between 95% LOA.
The study results indicate that the EarlySense MCPM device performs well when used to measure HR and RR as compared with the more invasive Masimo Rad-97 reference device. Advantages of the EarlySense device are that it is contactless and does not require disposable components. However, limitations include lack of MCPM when the neonate is not in bed, and lack of oxygen saturation monitoring. Notably, the Masimo Rad-97 capnography cannula was removed when the neonate was not in bed.
The accuracy of the EarlySense device was confirmed in a study population of high-risk adult surgical patients and in a small feasibility series with a paediatric and adult patient population in a sleep laboratory and intensive care unit setting.21 22 Other studies of monitoring technologies have been conducted in low-resource settings23–26; however, research conducted in neonates has largely been limited to studies in high-resource settings. Well-established methods for HR and RR monitoring exist, but limitations including inaccuracy, invasiveness, time-consuming application and high cost have prompted development of novel technologies.27–29 Novel contactless technologies are attractive as they avoid discomfort and risk of skin irritation. However, they have limitations of their own including motion artefacts, poor sensor coupling, poor-quality recordings and dependency on adequate lighting.28 29 There is a strong need for additional studies evaluating neonatal monitoring technology in low-resource settings.
Major strengths of our study were the independence of the investigators conducting the analysis, robust measurement of RR with high-resolution capnography, manual counting of breaths and randomised selection of minutes for comparison. However, our study site was better resourced than is typical in the region, and it is possible that the device may not have performed as well in a less-resourced environment. Currently, we are completing our next phase at Pumwani Maternity Hospital in Nairobi, Kenya, the largest referral maternal hospital in sub-Saharan Africa and a site more representative of a low-resource setting.
A substantial percentage (19.1%) of neonate recordings were not included for analysis due to data synchronisation and data identification issues (incorrect dates assigned). To minimise uncertainty in the reference device data and to avoid the EarlySense device missing the a priori threshold due to poor quality reference device data, we removed a substantial proportion of reference minutes from analysis and only included minutes with the highest quality reference data. In addition, we did not study clinical correlations or outcomes and were unable to assess the degree to which findings represented clinically significant differences. In the next phase of our study, we will assess the clinical feasibility of the EarlySense device with regard to accuracy, up-time, and event detection of high and low HR and RR events. Many of the neonates in this study were healthy, or relatively healthy, so it is possible that the agreement we observed may not reflect the sensor performance in critically ill neonates with higher or more irregular HRs and/or RRs. The EarlySense investigational device algorithms were developed in a cohort of neonates who were not critically ill to ensure that the initial evaluation and optimisation was performed in a more controlled environment, as is typically done for regulatory approval, before extending to more challenging cases and environments. We found the performance of this innovative non-invasive MCPM device to be promising in neonates. In a qualitative study, we found that healthcare administrators, healthcare providers and caregivers considered the EarlySense device to be feasible, useable and acceptable.30
Moving forward, it will be valuable to evaluate the EarlySense device for agreement with a reference device in neonates in intensive or critical care. A future study assessing the agreement of the EarlySense device HR and RR to ECG-derived HR and RR would strengthen the evidence for adoption of this novel contactless technology. In addition, evaluating the threshold and adaptive alerts provided by the device, and establishing cost-effectiveness, and clinical effectiveness will be necessary before large-scale implementation can be considered.
Data availability statement
Data are available in a public, open access repository. Data will be made available on completion of the secondary analyses of the data. We have ethics approval to deposit the data in an open access repository upon completion of the study.
Ethics statements
Patient consent for publication
Ethics approval
The research protocol was approved by the Western Institutional Review Board (20191102), Aga Khan University Nairobi Research Ethics Committee (2019/REC-02) and Kenya Pharmacy and Poisons Board (19/05/02/2019 (078)).
Acknowledgments
We would like to thank Dr Walter Karlen who developed the breath detection algorithm that was used to analyse the CO2 waveform data. This study was supported by a grant from the Bill & Melinda Gates Foundation (OPP12303136) and additional funding from Save the Children Innovation Fund.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors ASG and JMA conceived of the study and ASG obtained the funding. WMM, ASG and JMA designed and provided oversight for the administration and implementation of the study. JC contributed to implementation of the study. DD wrote the application used to collect data from the reference device and processed data. WMM, RO and DC collected the data. JMA, DW and DD verified the underlying data and DW performed data analysis along with YSH who performed data analysis, signal processing and algorithm optimisation. RK devised primary conceptual ideas for algorithm optimisation. GZ provided statistical analysis guidance. DW, ASG and JMA wrote the draft of the manuscript with critical input from the coauthors. All authors edited and reviewed the final manuscript. JMA acts as guarantor.
Funding Bill & Melinda Gates Foundation and Save the Children Innovation Fund. Bill and Melinda Gates Foundation programme officers provided input to the study design.
Competing interests YSH and RK are employed by EarlySense. All other authors declare no competing interests.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.