Aim To investigate whether airway management and non-invasive ventilatory skills are retained after the Neonatal Life support (NLS) course.
Methods Candidates who attended and passed the NLS course were retested by two registered instructors using the NLS ‘airway testing sheet’ unannounced at 3–5 and 12–14 months after their NLS course. Prior to the test, they were also asked to complete a proforma, indicating their own assessment of their competence in being able to effectively carry out all the items used in the NLS airway test.
Results Sixty-seven candidates were tested at 3–5 months, 26 (39%) passed first time, 34 (51%) on retest and 7 (10%) failed. At 12–14 months, 43 were tested, 19 (44%) passed on first attempt, 22 (51%) on retest and 2 (5%) failed. At 12–14 months, more candidates exposed to more than five resuscitations per month passed first time compared to those who were exposed to less than one resuscitation per month (p=0.029). More candidates who were offered resuscitation training at 6 monthly intervals compared to at yearly intervals passed the test on their first attempt at 3–5 months (p=0.022). Self-assessment of competence was not different between candidates who passed and those who failed.
Conclusions This study suggests that skills when tested in a simulated scenario are highly likely to have deteriorated within a few months of attending the NLS course. There is a need for research to determine whether deteriorations in skills after the NLS, as assessed by simulation, correlate with deterioration of skills in clinical practice.
- Medical Education
Statistics from Altmetric.com
What is already known on this topic
Resuscitation courses utilise simulation, which supports experiential and reflective learning and prepares the practitioner to deal with real patients.
Knowledge and particularly skills may deteriorate as early as 3 months after a resuscitation course.
What this study adds
Skills when tested in a simulated scenario deteriorate within a few months of attending the Neonatal Life support (NLS) course.
It is not known whether deteriorating skills after the NLS as assessed by simulation correlates with deterioration of skills in clinical practice.
Resuscitation courses are run in many countries to train healthcare professionals in adult, paediatric and neonatal resuscitation to attempt at optimising standards of clinical practice in resuscitation management, minimising error and decreasing patient morbidity and mortality. A central feature of these courses is the use of a multidisciplinary approach to teaching and learning to replicate the multidisciplinary team involvement which occurs in real resuscitations.1 The courses utilise simulation, which supports experiential and reflective learning2 ,3 and prepares the practitioner to deal with real patients.4
The evidence in the literature suggests that it allows them to acquire the necessary skills to deal with ‘real-life’ situations in preparation for clinical practice. The degree of knowledge and skill acquisition following resuscitation courses may vary.5 The testing of practical skills is crucial to assess if someone is a competent practitioner. We have shown in a recent systematic review that knowledge, and particularly skills, may deteriorate as early as 3 months after a resuscitation course.6 This issue has not been studied with respect to the neonatal life support (NLS) course. We wished to investigate whether airway management and non-invasive ventilatory skills taught on the NLS course were retained by the candidate after successful completion of the course, and furthermore, to explore the relationship between their retention of skills and both their own assessment and that of their peers of their competence at performing these skills.
Recruitment of the participants for the study took place on NLS courses occurring at Liverpool Women's Hospital between May 2007 and March 2009. At the beginning of the course, the study was explained to the candidates and information sheets, consent forms and a proforma for the candidates to complete with their demographic details and the contact number of their line manager were distributed. All completed forms from those wishing to take part were then collected in the lunch break of each course. It was made very clear that taking part was an ‘opt-in’ process and any candidate that did not return a consent form at this time was not approached again. Candidates who worked outside a 50 mile radius of the course centre were not eligible for inclusion, owing to the difficult logistics and time constraints of visiting them at their base hospital in order to follow them up. No distinction was made between candidates doing the NLS for the first time and those recertifying as this was felt to be unlikely to affect the results (given that the time between a candidate's previous course and the current one was at least 4 years and that there is evidence to suggest that in other resuscitation courses, skills and knowledge have fallen to baseline up to a year later). Approximately 2 months following the course, the line manager of each eligible candidate was contacted and appointments made for retesting the candidate unannounced between 3 and 5 months after the NLS at their place of work, using the same airway test that they had been given at the end of the NLS course.
All equipment utilised in the test, excluding the resuscitaire, was taken to each centre. Upon arrival of the assessors, the candidates were released from their clinical work by their line manager and informed of the imminent test. They were then asked to complete a demographic data form and a ‘self-assessment of competence’ proforma on which they had to mark on a Likert scale (range 1–7) their own assessment of their competence in being able to carry out effectively all the items used in the NLS airway test. This was based upon the airway test assessment sheet. The candidate then carried out the airway test in the assessment conditions as on the NLS course without any prompting from the assessors.
Candidates who were successful on their first attempt were congratulated and informed that the same process would take place again at approximately 12–14 months after their initial NLS course. If candidates were unsuccessful, they were offered a mini ‘booster’ training session. Immediately after this, a further airway test was carried out (this process also occurs on the NLS course if the candidate fails the first time). If the candidate was unsuccessful the second time, they were informed of this and errors were discussed and corrected, but the candidate then exited the study and was not tested at a later date.
An overall pass rate of 86% was assumed after testing the first 15 candidates. It was calculated that if the sample size was 50 (completed both retests), a 95% CI for a single proportion (using the large sample normal approximation) would extend 10% from the observed proportion for an expected proportion of 86%. Therefore, if a pass rate of 86% was observed, then the CI would change from 76% to 96%, which was considered adequate for this study. If, however, the sample size was 40 (completed both retests), a 95% CI for a single proportion (using the large sample normal approximation) would extend 10.8% from the observed proportion for an expected proportion of 86%. Forty first and second retests were therefore considered the minimum number that could be accepted for this study. Study recruitment was therefore projected to continue until at least 40 retests had been obtained.
During the study period, 13 NLS courses took place at Liverpool Women's Hospital with 294 candidates attending. A significant drop-out was expected owing to reticence of some candidates wanting to take part in a research study when attending a potentially stressful course, the logistics of the study, and that (not uncommonly on an NLS course) the candidate's base can be a significant distance from the NLS course centre. Of the 167 candidates who consented and were eligible, 67 (40%) were tested at 3–5 months after the course. Reasons for not testing at 3–5 months were living outside the testing area, incorrectly completed consent data forms, sickness, maternity leave and last minute change of shift. Out of candidates tested, 38 (57%) had undertaken the NLS previously (ie, were recertifying). Sixteen (24%) were midwives, 33 (49%) were nurses, one was a resuscitation officer and 17 (25%) were doctors. These proportions reflect the overall mix of candidates attending NLS courses, suggesting the sample of candidates taking part was representative of the overall population of candidates. Two candidates withdrew from the study within a week of attending the NLS course due to retirement prior to the retest date. Forty-three candidates were retested again at 12–14 months. The reasons for not retesting the other candidates were failure of first retest (7) or non-availability on the retest day due to change of job location or shift on retest day, sickness, line manager not contactable and maternity leave (17). Table 1 shows the numbers of candidates passing and failing the airway test at each time period after the NLS course and table 2 shows the reasons why the candidates failed.
The number of candidates passing the airway test on their first attempt at 3–5 and at 12–14 months after the NLS, according to number of resuscitations candidates are exposed to each month, are shown in table 3.
Significantly more candidates exposed to more than five resuscitations per month passed the first time compared to those who were exposed to less than one resuscitation per month. The frequency of resuscitation training updates candidates are exposed to and whether they passed the first time on retest at 3–5 months and at 12–14 months is shown in table 4. Significantly more candidates who were offered resuscitation training at 6 monthly intervals compared to yearly intervals passed the retest on their first attempt at 3–5 months.
Owing to the small numbers of candidates in the fail groups from the self-reported competence responses, overall median ‘confidence scores’ were calculated. For the seven candidates who failed at 3–5 months, the median confidence score was 6 (range 5–7). For the group who passed, the median confidence score was also 6 (range 5–6). There were some low scores in the larger group. Therefore, the data do not suggest that the overall confidence scores are lower in the group that failed the test. Among the 41 candidates that passed at 12–14 months, there was one data set for confidence scores not returned. The two median confidence scores in the two who failed were 5 and 7. For the group who passed, the median confidence score was 6 (range 5–6). There is little analysis that could be done on this data as the samples were too small, but the overall results were not inconsistent with the findings for the test at 3–5 months.
We have demonstrated a marked deterioration in both knowledge and skills as early as 3 months following attendance on the NLS course. This is the first time that this phenomenon has been demonstrated after the NLS course and confirms the findings of studies on other resuscitation courses.6 Loss of skills occurred despite recruitment taking place at a regional neonatal centre where candidates have more experience than others from smaller units. The small number of doctors attending the courses meant that comparison of outcomes between professional groups and grades of medical staff could not be meaningfully performed. The NLS courses taking place at Liverpool Women's Hospital are run according to nationally agreed standards,1 with the instructors teaching each aspect of the course in a standard way. This makes it likely that these results are generalisable to other courses and makes it unlikely that there were major differences between any of the 13 courses that candidates had attended, which could have affected the results of the study.
The NLS course has been designed to be educationally optimal through practical simulation experiences aimed at supporting experiential and reflective learning3 and incorporates many facets within the simulation scenarios that facilitate learning.4 Despite this, the study suggests that the ability to demonstrate knowledge and skills at a later date does not seem to be sustained. However, it can not necessarily be inferred that a candidate who did not ‘pass’ the airway test in this study would not be competent at resuscitation in real life. Essentially, they were being tested in a ‘surprise’ simulation—there could be other factors at play in a real resuscitation, not present in the simulated environment, which may result in a different performance in real life. The low-fidelity mannequin used for retesting may have made it more difficult for candidates to suspend disbelief compared to when attending the actual course, and following the ‘airway test’ algorithm might have been more difficult during this ‘one-off’ scenario where they had been taken unprepared from the clinical area compared to when they were ‘zoned-in’ to the simulation on the NLS course.
At 3–5 and 12–14 months post-training, the two main reasons for candidate failure on the airway test were changing the towel and auscultation of the heart rate, especially following the delivery of the inflation breaths. Given that this was a low-fidelity simulation that used dry towels and a mannequin, it is possible that candidates may be more likely in the clinical setting to remember to change a wet and soiled towel for a dry one. In the clinical setting, the assessment of the heart rate is usually made, as it is part of the Apgar scoring and may be more readily vocalised in a clinical setting.
Some studies have suggested that factors, which may ameliorate deterioration in knowledge and particularly skills, might be the provision of regular booster or refresher sessions and focusing on discrete skills as part of a task during training and at follow-up.7–12 On the NLS, all candidates who are unsuccessful on their first attempt are offered constructive feedback and a short booster training session prior to retesting. In our study, we offered candidates who failed on their first attempt a mini booster session immediately following the test. In brief, the airway opening manoeuvres available were discussed and candidates were informed of the reasons for them failing and allowed a brief time for practice prior to retesting. It is possible that this may have assisted in a further number of candidates passing on their second attempt. Frequent booster sessions for the individual may encourage a behavioural change by repetitive reinforcement. However, the effects of booster sessions after the NLS have not been assessed.
The current recommendation set by the UK Resuscitation Council is that health professionals repeat the NLS on a 4 yearly basis. Doctors who change posts every 6 months may be offered a booster training session at each induction at the start of their post, and mandatory NLS in-house training for resuscitation is often offered on an annual basis. Our study showed at 3–5 months and 12–14 months, the majority of candidates who passed the airway test on their first attempt had been exposed to resuscitation training on a 6 monthly basis, effectively receiving booster sessions, and supports the case for their effectiveness. Given this finding and a loss of skills evident after 3–5 months in our study, it therefore might be advantageous for a short refresher resuscitation programme to be offered between episodes of mandatory NLS training. Basing this on the NLS would allow for consistency which might not be present in in-house sessions. There is some evidence from a systematic review that these ‘in-house’ drills might improve clinical practice.10–12
Booster sessions have cost and human resource implications. It is unlikely to be practical to offer 3–6 monthly cycles of booster resuscitation sessions at institutions—rather it might be more feasible to embed aspects of deliberate practice (including resuscitation drills) at staff induction sessions and into daily work. Apart from further simulation sessions, other work has suggested that ‘reinforcement’ in the clinical area, to strengthen behaviour, will also improve competence.13 The results from the study suggest that, as perhaps might be expected, the more resuscitations candidates were exposed to in the clinical area, and the more likely they were to pass the airway test. For those candidates working in small district general hospitals, it is less likely that they will be involved in resuscitations on a regular basis compared to those working in a large regional unit. Staff in smaller units therefore may benefit from more in-house skills and drills sessions. One way of ensuring that the less exposed staff is receiving the skills and drills sessions could be for them to keep a formal record of resuscitations attended after the NLS course. After 6 months, unless a certain threshold number of resuscitations has been attended (eg, 5), a booster session could be offered in-house.
The literature has demonstrated that the use of videoing simulation is increasingly being utilised within medical training14 ,15 and has been used in research studies to assess knowledge and skill retention within the clinical setting.16 This could be used with ‘in-house’ booster sessions to improve their efficacy. Video assessment within the clinical setting may be a useful tool to assess the long-term retention of knowledge and skills. Videoing resuscitations, followed by a critical review of the team's actions, may also assist in acting as a ‘debrief’ for the staff involved. The logistics of this would include obtaining consent from mothers and staff to video them and the babies. As previously mentioned, clinical exposure to resuscitations is variable and depends greatly upon workplace, clinical role and opportunity; video assessment, therefore, may be particularly useful to those not carrying out frequent resuscitations, although there is no evidence to support this at present.
Previous studies have tested confidence at the end of resuscitation training, mostly in the form of a questionnaire or survey and have reported an increase in confidence immediately after.17–22 Our study showed little difference in the confidence scores for the candidates who failed and for those who passed at 3–5 and 12–14 months, suggesting that lack of confidence did not necessarily equate to lack of competence. Also—perhaps more importantly—presence of confidence does not always equate with competence.
In conclusion, skills and knowledge as assessed by low-fidelity simulation deteriorate significantly after the NLS course. It is possible that the structure and programme of the NLS could be redesigned to change education delivery.
The resuscitation theory could be taught as a ‘one-off’ training session, away from the clinical area as it does at present. This could be followed with shorter simulation sessions where each candidate practices and performs an airway test. The use of simulation with individuals working within their own clinical capacity could then be used to enhance the experience. These essentially would be booster sessions within the individual's clinical setting. The sessions would include NLS simulations, be based on a standard teaching package approved by the Resuscitation Council, be delivered by local accredited educators/instructors and occur on a frequent basis (every 4–6 months). These would be formally recognised as part of continuing professional development, the candidate being required to keep an up-to-date portfolio of achievements with each session being signed off by an instructor. This formative process would therefore be candidate-driven and could be more embedded into clinical practice as described above. It could improve resuscitation skills, inform instructors of weaker candidates who may need targeted training and ultimately improve patient safety.
The use of in-house ‘mock’ simulations on a weekly basis might also aid in the retention of knowledge and skills and act as a booster session. In these, healthcare professionals would be able to work and be assessed within their individual sphere of practice.
With thanks for the ‘Newborn’ Appeal for financially supporting this research and to the staff at the Liverpool Women's Hospital for their support.
Contributors BS wrote the study protocol, CMJM performed most of the simulation testing and CMJM and BS contributed in equal parts to the writing of the article.
Funding The Newborn Appeal Liverpool funded the salary for part-time research nurse.
Competing interests None.
Ethics approval Ethical approval was granted prior to the commencement of the study by The Liverpool Paediatric Research Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.