Increasing the Impact of Simulation Training during Covid-19 Limitations: A Technology-Based System for Increasing Trainee Engagement and Providing Real-Time Feedback

Background and Purpose: Enhanced learning methods, such as simulation labs, are essential in helping workers practice targeted skills in safe and controlled environments. Simulation labs vary in many aspects. Importantly, assessing competency demonstration during the simulation is often missed, especially regarding impacts of the simulation on trainees who are observing. Covid-19 restrictions have only further exacerbated the difficulty of assessing both participating and observing trainees. To help address this issue, this study explored the feasibility and impact of developing a feedback system that could be used for in-person or online delivery of simulation training. The system enables not only trainees who are participating but also trainees who are observing to be assessed on skill acquisition at multiple points during a simulation. Feedback is based on quantitative assessments, using ratings from expert trainers, and provided in real-time. The following questions guided our inquiry:

1. To what extent do observers recognize the target behaviors during simulation?
2. What impact does real-time feedback have on the perception of knowledge after simulation compared to a pre-assessment perception of knowledge?
3. What impact does real-time feedback have on the perception of skill after simulation compared to a pre-assessment perception of skill?
4. How effective do participants feel the simulation training format of real-time, quantitative feedback is?
5. What impact does the in-person vs. virtual meeting platform have on recognition of the target behaviors during simulation?

Methods: Each trainee completed real-time digital rating forms indicating whether each skill was effectively demonstrated for every section of the simulation lab. Then feedback was provided in real-time using concurrent expert trainer ratings on the same forms and expert observations of simulation performance.

A pre/post format was used to measure changes in perceptions of knowledge and skill in trainees. These data were correlated with self-efficacy reports from each observer after the simulation to explore the degree to which trainee performance correlated with self-efficacy. Differences in trainee performance and perceptions of learning effectiveness between in-person and online delivery formats (pre- and post-COVID-19, respectively) were analyzed.

Results: Trainee perceptions (N=300) of knowledge and skill significantly increased from pre to post. Findings suggest this assessment of peer observers during simulation increases engagement in the simulation lab and provide a more quantitative understanding of whether observers are learning to recognize when best practice skills are demonstrated. Of particular interest, moving the simulation to an online meeting platform did not negatively impact these results. All indicators of knowledge, skill, confidence, and perceived effectiveness remained relatively level post-Covid-19 compared to in-person simulations before.

Conclusions and Implications: This simulation method of increasing observer engagement and learning assessment enables trainers and educators to provide meaningful feedback to simulation participants who are observing, whether in-person or virtual. Such specific and quantitative feedback enables the broader effect of simulation beyond the individual learner “on the stage.” Impact on the transfer of learning will be explored in future studies.