Defining the Nature of Augmented Feedback for Learning Intraosseous Access Skills in Simulation-Based Health Professions Education

In the field of health professions education, acquiring technical skills involves three stages: 1) receiving instructions, 2) engaging in practice, and 3) receiving feedback. Simulation serves as a valuable tool that encompasses all three stages, enhancing the effectiveness of health professions education. This work focuses on feedback, which can be categorized as intrinsic (perceived by the learner through their senses) or augmented (provided by an external perspective). Augmented feedback can take the form of knowledge of results (information regarding the outcome) or knowledge of performance (information about the actions leading to the outcome). The overall objective of this work was to evaluate the perceived efficacy of these types of feedback in learning technical skills using a simulation, specifically an intraosseous access simulator, among advanced care paramedics. The primary focus of this article and the initial step towards achieving the aforementioned objective of this work was to determine the possible knowledge of results and knowledge of performance that paramedic facilitators could offer to advanced care paramedics during the use of an existing intraosseous access simulator. This research was conducted following the design-based research framework, employing a combination of design thinking and Delphi methods to generate a comprehensive list of augmented feedback, in both the form of knowledge of results and knowledge of performance, that can be provided to advanced care paramedics while learning intraosseous access skills through a simulator. The design thinking session was carried out to generate an initial inventory of augmented feedback, which was then refined through two rounds of Delphi consensus-building with paramedic experts. This process resulted in an eight-step list of feedback for knowledge of results and knowledge of performance that can be delivered to advanced care paramedics by paramedic facilitators using an intraosseous access simulator.


Introduction
Psychomotor skills are movement tasks that contain both cognitive and motor processes. These processes, in turn, often lead individuals to learn and manipulate the environment around them [1]. In the context of health professions education (HPE), psychomotor skills are often referred to as technical skills, being tasks that are performed by health professionals for the patient [2]. In HPE, learning technical skills involves three stages that ensure that both the cognitive and motor elements are understood: 1) instructions, 2) practice, and 3) feedback [2][3][4]. Instructions comprise the information provided to the learners about the skills and mechanical principles that are needed to perform the task [1]. Practice is an active process in which an individual attempts to perform a task with the intent of acquiring a new skill so that there is a permanent change in habit [1]. Finally, feedback can be defined as information that is given before, during, and after an action has occurred [1]. Feedback is typically divided into two categories: 1) intrinsic feedback and 2) augmented feedback. Intrinsic feedback is defined as the information that is naturally understood and gathered through our senses [1]. For example, a basketball player can see if they made their shot or feel if their throw was off. Augmented feedback supplements intrinsic feedback by providing information about the movement sequence or outcome from an external point of view [1]. Augmented feedback can be further subdivided into two main dimensions: knowledge of results (KR) and knowledge of performance (KP). KR is the feedback that is given regarding the outcome of the overall goal of the action. Using the same basketball example, the player's coach may provide feedback on the parabola of the ball after it was shot or where the ball hit the backboard to provide an explanation as to why the ball did or did not go into the net. KP is the feedback provided regarding the action movement that leads to the outcome [1]. This is where the basketball coach would provide feedback regarding the form of the player while they were making the shot. scripts to create an immersive, replicable, and standardized learning environment [5]. Therefore, using SBHPE to amplify HPE can be an effective tool that encompasses all three stages of learning technical skills.
Concerning feedback, it is unknown which type is most effective, especially within the context of SBHPE with advanced learners [1]. Therefore, the overarching aim of this work was to assess the perceived effectiveness of KP, KR, and intrinsic feedback in learning technical skills with advanced care paramedics (ACPs). We have selected intraosseous (IO) vascular access because it is a commonly utilized skill by ACPs, it has clear procedural steps, and we had access to a previously developed IO access simulator [6]. Before the perceived effectiveness could be investigated, we first had to determine what KP and KR could be provided specifically for learning IO access using the developed IO access simulator. Therefore, the objective of this article is to share the methods utilized and the results of determining what KP and KR can be provided in the context of learning IO access by ACPs.

Materials And Methods
The development of an advanced IO access simulator using 3D printing and silicone work was described in a previous report [6]. It is this IO access simulator that will be used, alongside feedback from a quality and development facilitator with an ACP certification, to assess the perceived effectiveness of KP, KR, and intrinsic feedback by ACPs from the Region of Durham Paramedic Services (Whitby, Ontario). However, before the perceived effectiveness can be investigated, we had to determine what KP and KR can be provided in the context of learning IO access using the advanced IO access simulator. The ethics for this project was exempted by the Ontario Tech University Research Ethics Board as per Tri-Council Policy Statement (TCPS2) Article 6.1 Issue Number 16892 as a protocol development study. The study as conducted at the Region of Durham Paramedic Services building (Whitby, Ontario). This work will be accomplished following the design-based research (DBR) framework, specifically within phase 1, which is the design phase. This phase will be accomplished using a hybrid of design thinking and Delphi methodology. Shown in Figure 1 below is how the methods will be situated within the DBR framework, with the phases in the green boxes being the phases utilized in this work.

Design-based research
In order to determine what KP and KR can be provided to ACPs when learning IO access using our previously developed IO access simulator, we followed the DBR framework. DBR is an educational framework that explains an iterative process focusing on the collaboration of researchers, stakeholders, and end-point users to generate solutions (i.e., resources) that can be applied to specific learning contexts [7,8]. DBR contains four iterative phases: 1) design, 2) test, 3) evaluate, and 4) reflect [9]. The design phase involves developing a solution that addresses both the theoretical and practical concerns of the problem [8,10]. The test phase involves implementing the solution in a real-world setting [11]. The evaluate phase evaluates the effectiveness of the solution using evidence from endpoint users' learning [12,13]. Finally, the reflect phase involves a retrospective analysis of the DBR methodology and methods used in the prior phases [12,14]. To achieve the desired outcomes, this work focuses on the design phase ( Figure 1). The methods used to accomplish this phase are a hybrid of design-thinking (DT) and Delphi.

DT and Delphi hybrid
The combination of DT and Delphi used to achieve the objective of this work is pictured in Figure 1. During the DT session, the define stage was conducted to determine what the objectives of learning IO access are, and the ideate stage is where we outline what augmented feedback, in the form of KP and KR, can be provided to learners when using the IO access simulator. The feedback determined from the ideate stage of the DT session was then put through Delphi rounds in order to gain consensus on what KP and KR can be provided to ACPs when learning IO access using the IO access simulator.

DT process
DT is a flexible and collaborative problem-solving process consisting of five stages: 1) empathize, 2) define, 3) ideate, 4) prototype, and 5) test [15]. The main goal of the empathize stage is for the researchers to get an understanding of who the stakeholders and end-point users are [15]. The define stage aims to pinpoint the needs and problems of the stakeholders and end-point users [15]. The ideate phase involves brainstorming possible solutions to the problem, which are then created in the prototyping phase [15]. The final test phase involves testing the proposed solutions to see if they meet the needs of endpoint users and stakeholders [15]. To fit into the scope of this work, we will just be focusing on the defining and ideating stages of DT. The DT session was held in person at the Region of Durham Paramedic Services building (Whitby, Ontario) with two members of the research team (Julia Micallef and Adam Dubrowski), a total of four participants (n=4); one ACP instructor (n=1), one ACP student (n=1), and two working ACPs (n=2). These participants were recruited via emails from the research team asking for participation. The inclusion criteria were that the participants had to be either a working ACP or an ACP student and had to be familiar with the IO access procedure. The one-hour session was conducted in the format of a focus group interview, facilitated by one of the researchers (JM), and guided using a PowerPoint presentation. The outputs from these phases then feed into the Delphi process.

Delphi process
The Delphi process is a structured group communication process that seeks to gather information and achieve consensus from a panel of experts using iterative survey questionnaires [16]. In general, the Delphi method process starts with the selection of a group of experts based on the topic being examined, referred to as the Delphi panel. Once all panelists are identified and confirmed, each is sent a survey with instructions to comment on each topic based on their personal opinion, experience, or previous research. They are then asked to return the surveys to the researcher, who groups the comments and prepares copies of the information. An agreement between the panelists is determined a priori, and the researcher determines if the agreement has been reached or if the content needs to be modified and resent for further deliberation by the panel. This is known as a Delphi round, and these rounds are repeated as many times as necessary to achieve a general sense of consensus [16]. For this work, we used an electronic survey (Google Forms), which was emailed to nine participants (n=9) consisting of ACPs from the Region of Durham Paramedic Services (n=6), paramedic educators (n=2), and a medical doctor (n=1). These participants were recruited via emails from the research team asking for participation. The inclusion criteria were that the participants had to be familiar with the IO access procedure. The survey was formatted so that each participant had to rate, on a 5point Likert scale, the level of importance of the steps in the form of KP and KR indicated in the DT session. In addition to participants ranking on a 5-point Likert scale, there were also sections in the survey where they could provide comments. The cutoff criteria for the Delphi methods were set a priori, where a median above 3 and a standard deviation below 1 would be considered a consensus among the participants to keep the particular item on the list. Items with a median lower than 3 and a standard deviation lower than 1 would be rejected, and items with a median lower than 3 but a standard deviation higher than 1 would be revised based on comments and included in subsequent rounds, with one week to complete each round. There were no limits to the number of rounds, and the end of the consensus-building exercise was reached when all items scored a median over 3 and showed a standard deviation lower than 1. For this work, only two rounds were needed.

DT session
The definition stage of the DT session was guided by asking the participants what the objectives for learning IO access are. This resulted in five main objectives: 1) to identify indications; 2) to use the appropriate tools; 3) to landmark correctly; 4) to ensure the IO needle is secured; and 5) to confirm the success of the IO access.
Next, during the ideation stage, the participants decided that the best course of action would be to use a previously developed skills checklist by one of the team members (DB), shown in Table 1, on how to perform an IO access and reword the steps so that they can be forms of augmented feedback (KP and KR). We condensed the 28 steps outlined in the skills checklist into seven steps that had a version of KP and KR for each step to ensure the feedback in each form was given in equal quantities. The results of the DT are shown in Table 2 below.  Step KP version KR version 1 The learner landmarks IO model 1-2 cm distal to tibial tuberosity on a flat portion of bone. The learner landmarked. 2 The learner cleans the intended injection site in a circular motion out from the injection site using an alcohol wipe. The learner disinfected the intended injection site. 3 The learner inserts the needle at approximately 90 degrees from the joint and stops when the bone is reached. The learner inserted the needle. 4 The learner drills into the bone until "pop" is felt. The learner drilled into the bone.

5
The learner secures the needle with a stabilizer so that there is no movement. The learner stabilized the needle. 6 The learner attaches a 10 ml syringe filled with saline to the needle and aspirates for blood and particles of marrow.
The learner aspirated.

Delphi rounds
The seven steps identified in the DT were then subjected to consensus-building exercises following the Delphi methodology [16]. The results of this first Delphi round are shown in Table 3 below, where the frequencies, median, and standard deviation are noted.
Step KP / KR  Based on the data from the first Delphi round, step six for both the KP and KR versions did not reach consensus (standard deviations were both above 1) and had to be fixed, as did the KP version of step seven since it did not meet the consensus criteria of having a standard deviation below 1. Comments from the participants are shown in Table 4. They indicated that aspirating for blood (step six) is no longer used in practice, so that step should be removed. They also noted that the KP for step seven was not worded properly and should be rewritten. Additionally, many participants noted that adding the step of preparing the patient as well as confirming the success of the procedure should be included as feedback as well.
Step Comments 1 KR step one -I believe it should say "successfully landmarked" as it is possible to landmark incorrectly.
2 Circular motion is best practice but may not represent exactly what is occurring in the field.

3
Again, I believe 'where' the learner inserted the needle is a key component of the KR.
Not a fan of the wording of the KP statement. Should read: "The learner inserts the needle through the skin at a 90-degree angle to the bone and stops when the bone is contacted." KR could be done incorrectly.

4
KR -Drilled into the bone in the correct area, ending in the osseous space without going through the bone.
The KP and the KR statement are more than just drilling into the bone. The result should be the needle is drilled into the bone with consistent pressure and released when the pop is felt.
KR may not have reached the correct location.
The pop may not always be felt. Resistance is the key -resistance will lighten once in.

5
There really should be a step between four and five where the stylet is removed from the needle in the bone.
Possibly the same result if the appropriate stabilizer device is used.

6
Aspirating for blood is no longer in our step-by-step process for confirmation, although it does confirm placement.
Depending on where you read, this step may not be necessary. Recent procedures have steered away from this, as sometimes the bone marrow can block the IO needle.
FYI, if a 10 ml syringe is filled with saline, there is no room to aspirate any material as the syringe is already full. The syringe could actually be empty for the purpose of aspiration.
Doesn't always yield a positive result when done.

7
Although this is important, your initial flush is used to confirm patency and placement by determining if you 'don't' feel infiltration into the surrounding tissues. This might be hard in a simulated environment. You may be able to do this by creating a reservoir where saline can collect if the placement is not correct.
The "flush" should be relatively quick to create the open space that allows the IV to drip afterwards. This is correct but can be worded that a saline flush is administered to ensure patency which is evidenced by the flow of liquid from the IO trainer.
Not sure I understand this fully, but sounds as though this is to confirm the site, so it would be more important for the result than the actual skill of slowly infusing saline which is not part of IO insertion in practice Overall I think this is a minimal difference between the KP and the KR for a skill like this.
Consider adding a final step speaking to confirm the IO procedure by ensuring the IV runs well, no signs of infiltration, no bruising or swelling of the leg etc.
Maybe something related to prepping the patient (e.g., positioning).
These all seem appropriate and would suggest technique and result are ideal for most of these steps for learning.
You could have put something about indicating this procedure. These results were then incorporated into the new steps listed in Table 5 below. In this new list of steps, step six was removed, two new steps were added, and the KP version of step seven was fixed, resulting in eight steps in the form of KP and KR. Only the new steps (steps one, seven, and eight in Table 3) went through another Delphi round with the same participants from round one and were given one week to complete. The results from the second Delphi round are shown in Table 6.   The new step one did not reach consensus for both the KP and KR versions due to the standard deviation being above one. The comments are shown in Table 7 below, and they indicated that the step is important to indicate that the IO access simulator is positioned properly; however, indicating placement is not important as the simulator will already be on a steady surface; therefore, the step was reworded for KP and KR to satisfy the comments. The remainder of the steps met with consensus.
Step Comments I think the placement/display of the IO model on a sturdy surface may potentially impact the success of the remainder of the skill, however seeing as this "perfect" scenario never exists in the field, I don't think it's a valuable measure. Perhaps a more realistic scenario would be beneficial.
It is important to comment on the external rotation as it will be difficult to perform the skill if the model isn't in the right position.
The student must place the IO model on a hard surface so that he can train efficiently and safely.

7
I agree with your statement and understanding that saline access is used as placement confirmation. While other factors like feeling the "pop" or resistance met are confirmation methods directly related to the skill of IO insertion, I think this would be an important factor for an expert to evaluate the overall success of the skill performed. My other comparison would be the skill of intubation, and attaching equipment to see if lungs inflate -not directly related to the skill itself, but an important factor to evaluate. I hope that makes sense.
I agree with the comments stating the flush is required to ensure patency and avoid infiltration into the surrounding tissues.
The KP doesn't sound right to me, I believe it should state: The learner administers a saline flush and assesses for infiltration into the surrounding tissues. In real life, we can't see the fluid flow from the end of the model and therefore the KR would confirm you do not have infiltration (fluid in the surrounding tissues and not in the osseous space).
It is important for the student to understand that this step (injecting saline and monitoring the drip) is important as it is part of successfully performing the procedure.

8
Typically the fluid would be placed under pressure.
This is a good additional step -as it is a key component of initiating the IO.
Step eight, we are actually looking at flow rate and it would be more important to look at the rate of flow because you may need to make a decision to use a pressure infuser to actually get a steady flow rate.
It is important for the student to understand that this step (injecting saline and monitoring the drip) is important as it is part of successfully performing the procedure. The resulting steps from the Delphi rounds are shown in Table 8. It is these steps that will be used by instructors to provide learners with KP and KR in the next study of this research to assess the perceived effectiveness of KP and KR on learning IO access using an IO access simulator.

Discussion
This discussion is organized around two main contributions: 1) pragmatic and 2) methodological. First, the purpose of this study was to determine what KP and KR can be provided to ACPs when learning IO access using an IO access simulator. To accomplish this, we followed the initial phase of a DBR approach, specifically within the design phase, using a hybrid of DT and Delphi methods. The main purpose of the design phase of DBR is to generate solutions that can address the theoretical and practical implications of a specific learning problem [8,10]. The theoretical concern we are addressing in this work is the matter of what type of feedback is perceived to be most effective by ACPs when learning IO access using an IO access simulator. When learning technical skills, such as IO access, feedback is the most important feature; however, it is still not known whether KP or KR is more effective [1,17]. Practically, we needed to determine what KP and KR can be given to learners, specifically in the context of learning IO access in an SBHPE environment. The DT session cultivated an initial list of steps, with each step being feedback that can be provided in the form of KP as well as KR. The Delphi rounds refined the list to result in eight steps, each written in the form of KP and KR so that it can be used as a guide for paramedic instructors to provide augmented feedback to ACPs learning IO access using an IO access simulator. In summary, the result was a set of steps when feedback needs to be provided. For each step, the experts and the learners provided input on how to operationalize the feedback to be either KR or KP in nature. This list will be used in subsequent research that will focus on testing and evaluating the perceived effectiveness of these types of feedback in training.
Second, the methodological contributions of this paper are centered around the use of the DBR approach in the construction of instructional (i.e., feedback) materials in SBHPE. Specifically, DBR was introduced to the educational field in the early 1990s by Ann Brown with the purpose of creating interventions in a collaborative manner (between the researcher and practitioner) that can be used in educational settings [13]. While this methodology has been predominantly used for traditional learning settings, with the increase in SBHPE, DBR can be applied to experiential learning as well. In an article by Schmitz et al. [18], a mobile simulation game was successfully designed and implemented following a DBR approach. Additionally, DBR has been used to create educational models for simulation facilitators [19]. There are no requirements for the methods used in each of the phases of DBR [9]. Therefore, using a combination of DT and Delphi methods to accomplish the design phase of DBR is unique to this research. However, it is not unique to SBHPE, as a study by Sivanathan et al. [20] has used a combination of DT and Delphi to help guide the design of a virtual reality simulation to help with moral distress experienced by healthcare professionals. As DBR necessitates the cooperation of researchers, designers, educators, and learners, employing a blend of DT and Delphi methods offers a chance to fulfill the requirements of all involved parties. For example, designers, educators, and learners prefer utilizing DT, a creative design process that facilitates idea generation and problemsolving. Conversely, the use of the Delphi method satisfies the rigorous demands of researchers seeking approaches that ensure content validity. Therefore, using a combination of the two approaches allows for the creative generation of solutions that can be validated through consensus-building [21].
There are a few limitations and strengths to this study. First, while DT and Delphi methods are validated methodological approaches, using them in combination with each being modified has not been validated as an approach. Despite this combination being used successfully in the past [20,21] for SBHPE scenarios, the approach itself has not been tested and validated. On the other hand, this methodological combination is unique to DBR and can be an advancement for the field. One of the strengths of this work is that we were able to get the perspectives and feedback of stakeholders (paramedic instructors) as well as end-point users (ACPs and ACP students). However, our small sample size for the Delphi rounds was a limitation, as a sample of 15-30 is more adequate [16].

Conclusions
In this article, the utilization of the DBR framework integrating the DT and Delphi methods is described. The primary objective of this study was to determine what KP and KR can be provided by paramedic instructors to ACPs in the context of learning IO access skills using a previously developed IO access simulator. Through the systematic application of the DT method, an initial list of feedback in the form of KP and KR was developed based on an IO access skills checklist. The Delphi method provided a consensus on the identified KP and KR for instructing ACPs in IO access skills. This resulted in an eight-step list of feedback in the form of KP and KR, which will be used in the next phase of this research project, which assesses the perceived effectiveness of feedback in training ACPs IO access skills in an SBHPE environment.

Additional Information Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. Ontario Tech University Research Ethics Board issued approval 16892. The ethics for this project was exempted by the Ontario Tech University Research Ethics Board as per TCPS2 Article 6.1 Issue Number 16892 as a protocol development study. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.