Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience

Caroline Taylor; Adalia Ikiroma; Anne Crowe; David H Felix; Gillian Grant; Lucy Mitchell; Teresa Ross; Margaret Saunderson; Linda Young

doi:10.1038/s41405-023-00132-0

Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience

Taylor, Caroline; Ikiroma, Adalia; Crowe, Anne; Felix, David H; Grant, Gillian; Mitchell, Lucy; Ross, Teresa; Saunderson, Margaret; Young, Linda 2023-02-07 00:00:00 www.nature.com/bdjopen ARTICLE OPEN Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience 1✉ 2 1 3 1 1 1 Caroline Taylor , Adalia Ikiroma , Anne Crowe , David H Felix , Gillian Grant , Lucy Mitchell , Teresa Ross , 1 2 Margaret Saunderson and Linda Young © The Author(s) 2023 AIM/OBJECTIVE: This study evaluates the effectiveness and users’ experience of using live stream technology to conduct workplace observation assessments of trainee dental nurses. Information on the usability, accessibility, and general satisfaction of this technological technique were collected. MATERIALS AND METHODS: This nationwide cross-sectional survey was conducted in Scotland and included one focus group and three online questionnaires with qualitative and quantitative questions. The quantitative responses were described using standard descriptive analysis, while the quantitative data were investigated using thematic analysis. RESULTS: Eighty-one trainee dental nurses, 35 clinicians and 19 assessors participated in this study. Live stream observation was generally well received by the trainee dental nurses and clinicians, who thought that it had helped increase their conﬁdence to perform practical skills. The assessors also stated that overall satisfaction was high, and that live stream observation met their expectations for efﬁcacy. However, several technical challenges, such as network issues were brought up by responders. CONCLUSION: This study provides evidence that workplace observation assessments can be performed in the future by using live stream technology. However, additional investigation and comparison will aid in determining the most effective way of using this approach and providing feedback to promote learning among dental trainees. BDJ Open (2023) 9:4 ; https://doi.org/10.1038/s41405-023-00132-0 INTRODUCTION Dental Nurses are members of a group of Dental Care The coronavirus disease-19 (COVID-19) pandemic resulted in Professionals (DCPs) who play a vital role within a dental team global lockdown and created disruptions in educational delivery and provide support and assistance in clinical and non-clinical and assessment processes. Many educational activities including aspects in providing high-quality patient care. All dental profes- course completion examinations and certiﬁcation were either sionals must be registered with the General Dental Council (GDC) postponed or cancelled. Of particular concern was the delivery of before they can work in the United Kingdom [8]. It is a robust and timely observational assessment of practical skills [1]. requirement that dental nurses undertake a pre-registration Assessment is regarded as a crucial component in the learning training programme with a training provider approved by the process [2], and it should be sustainable if it is to foster lifelong GDC towards GDC recognised qualiﬁcation. Mindful of this, Skills learning [3]. This suggests that no matter what assessment for Health recognised that assessors may ﬁnd it more difﬁcult to method is utilised, it should effectively stimulate engagement to gain access to a dental environment to carry out observations improve learning and performance, and thus promote sustainable during COVID-19 restrictions and authorised the use of live video development of learning capabilities. Emphasis on workpla- stream as a suitable assessment method [9]. Skills for Health is a ce‑based assessment (WBA) in dental health professionals’ training non-proﬁt organisation committed to the development of an has been increasing in the last decade [4–7]. Most WBAs are improved and sustainable healthcare workforce across the United observational and help the assessor to assess a trainee’s Kingdom. professional competence in a range of clinical scenarios. They Observation of practice is a primary source of evidence and a also provide the opportunity for timely and objective feedback to fundamental principle of the Scottish Vocational Qualiﬁcation the learner about both their clinical and non-clinical soft skills [7]. (SVQ) in Dental Nursing qualiﬁcation [10]. The SVQ standards A key aim of feedback is to encourage and support learners to require assessment methods to be selected on the basis of their self-assess and reﬂect on their learning to help them self-correct internal veriﬁcation process, which is made up of its validity, and orient their current performance. reliability, and practicability [11]. The validity of an assessment is 1 2 Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK. Dental Clinical Effectiveness Workstream, NHS Education for Scotland, Edinburgh, UK. Dental Directorate, NHS Education for Scotland, Edinburgh, UK. email: Caroline.taylor@nhs.scot Received: 8 November 2022 Revised: 22 December 2022 Accepted: 13 January 2023 1234567890();,: C. Taylor et al. the degree to which the assessment method is appropriate to the METHODS standards. The reliability of an assessment is the degree of Design This was a national cross-sectional survey conducted in Scotland consistent results when used with different candidates, different comprising three online questionnaire and one focus group. assessors and on different occasions. The practicability of an assessment refers to the feasibility that the assessment makes best use of available resources, equipment, and time. Prior to the Participants COVID-19 pandemic, in order to conduct a WBA all assessors were The study participants included all trainee dental nurses undertaking pre- registration training with NES, the clinicians they provided chairside required to travel to the trainee’s workplace. Depending on the support to and assessors who completed the live stream observation geographic distance from the assessor’s base to the trainee’s assessment process. The clinicians were dentists, dental therapists, and workplace, conducting the observation might result in at least two hygienists who work with the trainees in their practice, and the assessors visits which made a signiﬁcant impact on travelling time were Dental Tutors employed by NES, who carry out the role of an and costs. Assessor. El-Kishawi et al. highlighted that evolving technologies and methods of assessment could enhance students’ learning environ- Description of the Assessment Process ments and improve tutors’ assessment experiences [2]. Some studies Evidence of candidates’ performance is drawn primarily from naturally have reported students’ positive learning experience from using occurring work activities that take place under normal working conditions digital technology for assessment in both medical and non-health in a normal work environment. This WBA requires an assessor to observe a disciplines [1, 12–16]. These studies have uncovered beneﬁts at the trainee dental nurse while carrying out practical based tasks as deﬁned in tertiary education level such as increased access and ﬂexibility [1], the qualiﬁcation standards as they provide chairside support to a GDC the provision of a platform for an interactive and independent registered dentist, hygienist, or therapist providing care to the patient. This learning opportunity [15], a useful tool for improving practice and is referred to as a Performance Criteria, and it describes the level of competency required for the SVQ in Dental Nursing qualiﬁcation. This assessing competency [12], and the opportunity to develop skills in observation is conducted in the trainee’s own workplace, which could be a the use of computer technology [13]. Additional beneﬁts beyond General Dental Practice, Public or Hospital Dental Service. Students skill acquisition were identiﬁed in a recent study which found that pursuing the SVQ in Dental Nursing must be observed supporting the video feedback promoted greater knowledge retention, identiﬁed patient and the clinician as they provide chairside support during a variety performance situation, clinical reasoning, self-efﬁcacy, and learning of clinical dental procedures and duties within their role. The assessor satisfaction experience among students [17]. The outcome from records their performance and provides feedback on their ﬁndings. This video feedback was greater compared to computer assisted provides evidence for the trainee’s portfolio to demonstrate achievement. assessment analysis report or online-written comment. Assessors must be competent in the area of practice to which the speciﬁc To ensure uninterrupted progress and reduce the impact of the national occupational standards apply, according to the Scottish Qualiﬁca- tions Authority (SQA), and for the SVQ in Dental Nursing, assessors must COVID-19 pandemic restrictions on trainee dental nurses, NHS hold a qualiﬁcation recognised by the regulatory body for enrolment and Education for Scotland’s (NES) DCP Workstream devised the process maintain professional registration with the GDC as well as obtain an to align the Skills For Health COVID-19 supplementary guidelines for assessor qualiﬁcation. the assessment strategy. With reports that the use of real-time Microsoft Teams Platform (https://www.microsoft.com), a workspace for online assessment has good prospects [1] and culminates in positive real-time collaboration and communication was used during the assess- academic outcomes [18], a process utilising live video stream to ment. Microsoft Teams is freely available for download, with secure conduct observational WBAs and effectively assess performance was connection, and features that allow multiple participants and screen designed and applied. Research exploring the use of digital sharing. The meeting link was shared with all participants, allowing them technology has investigated the reliability and accuracy of live to connect, the assessor to observe, and interact during the practical observation. Before the observation, a test run was conducted to ensure stream observation for performance assessment and feedback; connectivity and video/audio quality. [1, 19–23] however, research evaluating the use of live stream technology at workplace level is in its infancy. Wang et al. reported its usefulness and feasibility with high interrater reliability of video- Questionnaire Development based assessment of medical residents’ performance [21]. McQueen There were three questionnaires developed for data collection: one for dental nurse trainees, one for clinicians, and one for assessors. The majority et al. highlighted the beneﬁts of video-based assessment including of the sections in the trainee and clinician questionnaires were the same. the ability to capture clinical ability in the operating room without The trainee questionnaire was divided into ﬁve sections to collect data on decreasing intraoperative efﬁciency and the potential to improve (1) preparation (seven questions), (2) access and usability (four questions), formative assessment and feedback practices [20]. Similarly, Rao (3) equipment (two questions), (4) support and feedback (four questions et al. found that 85% of participants strongly agreed and 15% for clinicians and ﬁve questions for trainees), and (5) expectation agreed that essential minor deviations during real-time online [performance/overall satisfaction] (two questions). Both questionnaires practical assessment did not compromise the basic principles and included open-ended questions for comments on the aforementioned goals of assessment, and the quality of assessment through this measures. The assessor questionnaire included 3-items to collect informa- mode of assessment was graded as good to excellent [1]. Therefore, tion about expectations: [effectiveness/overall satisfaction] (two questions) and an open-ended question about effectively assessing trainee the evidence suggests digital technology can transform the way performance. workplace‑based assessment is delivered. Some of the closed-ended question items required a binary response However, very little is known about the use of technology in (Yes/No). Most questions were based on a 3-point Likert scale with ‘3’ performing workplace assessment in dental education. It is indicating never and ‘1’, all the time to statements in the access and important to consider whether it can be objectively measured usability domain; a 4-point Likert scale with ‘4’ indicating immediately and and evaluate users’ perceptions of its usefulness and effectiveness, ‘1’, several days later to statements in the feedback domain; and a 5-point particularly in the post-pandemic era. This knowledge is key to Likert scale with ‘5’ indicating strong agreement and ‘1’, strong integrating and sustaining the use of live stream technology in disagreement to statements in the preparation, support and feedback, vocational dental education programmes which are actively performance, effectiveness, and overall satisfaction domains, respectively. seeking opportunities to embed technology-based education to improve their programme. Therefore, the focus of this study was Data Collection to explore the views of trainee dental nurses, clinicians, and This was an online questionnaire survey hosted on Questback which is the assessors to evaluate users’ experiences and satisfaction of using NES approved online survey platform. This was distributed by email live stream technology to conduct workplace observational containing a secure link to the survey to the trainees and clinicians after completing the live stream observation assessment, whilst the assessors assessments. BDJ Open (2023) 9:4 C. Taylor et al. before the observation. Ninety-six percent agreed that they Table 1. Median rating and reliability according to the measures for followed the guidance for the observation, (93%) agreed that respondents. the guidance was discussed with their teams and (87%) agreed that their workplace provided support in preparation for the Category Measure x͂ IQR α/ρ observation. Eighty-seven percent agreed there were able to test Trainee 0.72 the process with an assessor before the observation and (89%) Preparation 5.00 4.00–5.00 0.63 agreed that their workplace provided support in preparing Access and usability 3.00 2.00–3.00 0.63 them for the observation. In addition, (96%) of the respondents Feedback and support 5.00 4.00–5.00 0.61 agreed that patients were comfortable with the method used for the observation, Table 2. Expectation 4.50 4.00–5.00 0.69 Clinicians agreed that their trainees were prepared for the live (performance and stream observation (median = 4.0 IQR 3.0-4.0; α = 0.65), Table 1. overall satisfaction) The majority (91%) also received the guidance document that Clinician 0.82 provided information on the method of assessment and vital Preparation 4.00 3.00–4.00 0.65 support for the trainees before the observation. Over three- Access and usability 3.00 2.00–3.00 0.80 quarters (77%) agreed that the instruction within the guidance was clear to follow. Eighty-three percent agreed that they Feedback and support 3.00 3.00–4.00 0.68 discussed the content of the guidance document with their Expectation 4.00 3.50–4.50 0.84 trainee in advance of the observation. In addition, 71% agreed (performance and that they were able to test the process with the assessor before overall satisfaction) the observation and 94% agreed that patients involved in the live Assessor Expectation 3.50 3.00–4.00 0.81 stream observation were comfortable with the method used. (effectiveness and Eighty-seven percent agreed that they felt comfortable with using overall satisfaction) technology for the trainee’s observations. The majority (91%) of Note. x-͂ Median; IQR- Inter-Quartile Range α/ρ-Cronbach’s alpha/Spearman- clinicians reported that overall, they felt that sufﬁcient support Brown coefﬁcient was received from the assessor in preparing for the trainee’s The value for equipment domain was not included in the table as the internal observation, Table 2. consistency was not satisfactory Access and usability Trainees agreed that the live stream observation was accessible completed their questionnaires at the end of the observation assessment period. Data were collected from November 2020 for a period of two and easy to use (median = 3.0; IQR 2.5–3.0; ρ = 0.63). Just under weeks to complete after the live stream observation assessment. In three quarters (73%) reported that they never had difﬁculties addition, invitations to participate in online focus groups via Microsoft accessing suitable equipment to undertake the observation. Most Teams were circulated to all assessors employed by NES. (89%) reported that they were able to access the live stream with ease all the time to undertake the observation. In addition, more Data Analysis than half of the trainees (64%) reported that they had reliable Standard descriptive analyses were performed to describe the responses connectivity all the time throughout the observation, and (69%) using the Statistical Package for the Social Sciences (SPSS) version 27 [24]. achieved clear view of the assessor to review their performance all For each, median (and interquartile range [IQR]) rating was used as central the time, Table 2. tendency (central quartile range) and internal consistency using Cron- Clinicians agreed that the live stream observation was accessible bach’s alpha or Spearman-Brown coefﬁcient. A cut-off value of 0.6 was and easy to use (median = 3.0; IQR 2.0–3.0; α= 0.80), Table 1.Around considered a good internal consistency of the items in the scale [25]. Items two-thirds of clinicians (66%) reported that they never had difﬁculties were reverse scored when necessary; therefore, the higher the score, the accessing suitable equipment to undertake the observation. More more positive the response. The open-ended data were explored using than half (54%) reported that they were able to access the live stream thematic analysis [26, 27] to identify themes which emerged from the data with ease all the time to undertake the observation. About half (51%) within the measures. This involved organising the qualitative data into themes and was performed on Microsoft Excel. reported that they had reliable connectivity all the time throughout the observation, and around two-thirds (66%) achieved a clear view of the clinical area for the assessor to view their trainee’s Results performance all the time, Table 2. Out of 81 trainees that participated in the live stream observation, 45 completed the questionnaire with a response rate of 55.6%. Additionally,19 assessors participated in the live stream observation, although only 15 of Equipment them, together with 35 clinicians, completed the questionnaire. The For the equipment measure, the Spearman-Brown coefﬁcient for response rate was 78.9% for assessors; however, it was not possible to both trainees (ρ = 0.36) and clinicians (ρ = 0.48) was <0.6, and calculate this for the clinicians as the total number involved in the trainee’s therefore an overall measure was not calculated. For the type of IT remote observation assessment is unknown. In line with the research equipment used for the observation, more than half of the question, the collected data provided information about the participants’ trainees (62%) and around two-third (66%) of clinicians used smart perspectives on the effectiveness and overall satisfaction of using live phone while (16%) and (11%) used laptop. In addition, more than stream technology to conduct workplace observation competency half (62%) of the trainees and around two-third (66%) of clinicians assessment of trainee dental nurses. The internal consistency coefﬁcient across all items for trainees, clinicians and assessors were 0.72, 0.82, and reported that they did not use additional equipment to hold or 0.81, respectively, Table 1. position their device to undertake the observation, while over a third 38% and just over a third (34%) used one to hold or position their device, Table 2. TRAINEE AND CLINICIAN QUESTIONNAIRE FINDINGS Preparation Feedback and support Trainees agreed that they were prepared for the live stream Overall, trainees agreed that they were supported for the live observation (median = 5.0; IQR 4.0-5.0; α = 0.63), Table 1. The stream observation (median = 5.0; IQR 4.0–5.0; α = 0.61), Table 1. majority (93%) received the guidance document that provided Roughly half, 47% of respondents agreed that the assessor information on the method of assessment and vital support provided technology/IT assistance when needed, 98% agreed that BDJ Open (2023) 9:4 C. Taylor et al. Table 2. Descriptive statistics for trainees and clinicians. Table 2. continued Measure/Questions Trainees Clinicians Measure/Questions Trainees Clinicians Expectation (performance and overall satisfaction) The feedback received: Was helpful in supporting development Allowed demonstration of performance as a dental nurse Agree n(%) 45 (100) 29 (82.9) Agree n(%) 45 (100) 29 (82.9) Uncertain n(%) – 5 (14.3) Uncertain n(%) – 5 (14.3) Disagree n(%) – 1 (2.9) Disagree n(%) – 1 (2.9) Provide any technology/IT assistance when needed Overall satisfaction of your experience Yes n(%) 21 (46.7) 14 (40.0) Excellent n(%) 20 (44.4) 8 (22.9) No n(%) 24 (53.3) 21 (60.0) Good n(%) 25 (55.6) 26 (74.3) Access and usability Poor n(%) – 1(2.9) Difﬁculties accessing suitable equipment needed for the observation Preparation All of the time n(%) 1 (2.2) 1 (2.9) Received the live stream observation guidance document Some of the time n(%) 11 (24.4) 11 (31.4) Yes n(%) 42 (93.3) 32 (91.4) Never n(%) 33 (73.3) 23 (65.7) No n(%) 3 (6.7) 3 (8.6) Able to access the live stream session with ease The instructions within the guidance was clear to follow All of the time n(%) 40 (88.9) 19 (54.3) Agree n(%) 43 (95.6) 27 (77.1) Some of the time n(%) 5 (11.1) 13 (37.1) Uncertain n(%) 1 (2.2) 7 (20.0) Never n(%) – 3 (8.6) Disagree n(%) 1 (2.2) 1 (2.9) Connectivity reliable throughout the observation Discussed the content of the guidance document All of the time n(%) 29 (64.4) 18 (51.4) Agree n(%) 42 (93.3) 29 (82.9) Some of the time n(%) 15 (33.3) 16 (45.7) Uncertain n(%) 2 (4.4) 5 (14.3) Never n(%) 1 (2.2) 1 (2.9) Disagree n(%) 1 (2.2) 1 (2.9) Able to achieve a clear view of the clinical area Able to test the process before the observation All of the time n(%) 31 (68.9) 23 (65.7) Agree n(%) 39 (86.7) 25 (71.4) Some of the time n(%) 14 (31.1) 9 (25.7) Uncertain n(%) – 6 (17.1) Never n(%) – 3 (8.6) Disagree n(%) 6 (13.3) 4 (11.4) Equipment Workplace provided support in preparing for the observation Type of IT equipment used Agree n(%) 40 (88.9) 32 (91.4) Android 3 (6.7) 1 (2.9) Uncertain n(%) 4 (8.9) – Desktop/PC 2 (4.4) 4 (11.4) Disagree n(%) 1 (2.2) 3 (8.6) Laptop 7 (15.6) 4 (11.4) Patients involved were comfortable with the method used Tablet 5 (11.1) 3 (8.6) Agree n(%) 43 (95.6) 33 (94.3) Smart phone 28 (62.2) 23 (65.7) Uncertain n(%) 2 (4.4) 2 (5.7) Additional equipment to hold/position device Disagree n(%) –– Yes n(%) 17 (37.8) 12 (34.3) Felt comfortable using technology for the workplace observation No n(%) 28 (62.2) 23 (65.7) Agree n(%) 43 (95.6) 31 (88.6) Different question for clinicians—Overall, did you feel that you received sufﬁcient support from the assessor in preparing for the trainee’s Uncertain n(%) 2 (4.4) 2 (5.7) observation? Disagree n(%) – 2 (5.7) Trainee: The feedback received was: Timely; Clinician: Did the trainee share their assessor feedback with you? Feedback and support How soon was feedback received after the observation the feedback from the assessor was timely, all respondents (100%) Immediately n(%) 27 (60.0) – agreed that the feedback from the assessor was clear and Same day n(%) 14 (31.1) – constructive and supported their development, Table 2. Next day n(%) 3 (6.7) – Overall, clinicians’ perspective towards feedback and support, Several days later n(%) 1 (2.2) – was good, (median = 3.0; IQR 3.0–4.0; α = 0.68), Table 1. There was an equal agreement (83%) among clinicians with the statement The feedback received: Timely / Shared the assessor feedback “feedback received was clear and constructive” and “feedback Agree n(%) 44 (97.8) 30 (85.7) received was helpful in supporting trainee development”. The vast Uncertain n(%) 1 (2.2) – majority (86%) of clinicians said the trainees shared their assessor Disagree n(%) – 5 (14.3) feedback with them. However, when asked if the assessor provided technology/IT assistance when needed, more than half The feedback received: Was clear and constructive (60%) of clinicians said ‘no’, Table 2. Agree n(%) 45 (100) 29 (82.9) Uncertain n(%) – 6 (17.1) Expectation Disagree n(%) –– Overall, trainees agreed that using the live stream observation met their expectations (median = 4.5; IQR 4.0–5.0; ρ = 0.69), BDJ Open (2023) 9:4 C. Taylor et al. Table 1. To evaluate trainees’ expectation based on perfor- CLINICIAN AND TRAINEE mance, all respondents (100%) agreed that the live stream Themes for Preparation observation allowed them to demonstrate their performance. Both trainees and clinicians reported similar themes that For expectation based on overall satisfaction, nearly one-half enhanced their preparation for the live stream observation. Four (44%) reported their experience as ‘excellent’ and 56% had a distinct themes were identiﬁed that describe what it meant to ‘good’ experience from using the live stream observation for participants to be prepared for the observation process. These their assessment, Table 2. four themes are sense of responsibility, being organised, inﬂuence of Overall, clinicians agreed that using the live stream observa- time, and team collaboration. The participants described how tion met their expectations (median = 4.0; IQR 3.5–4.5; ρ = 0.84), important these themes were with regards to the type of support Table 1. To evaluate clinicians’ expectation based on perfor- provided by their team and their preparedness for the mance, The majority (83%) of clinicians agreed that the live observation. stream observation allowed their trainees to demonstrate their performance. For expectation based on overall satisfaction, just Sense of responsibility. The ﬁrst theme, sense of responsibility was under three quarters (74%) of clinicians rated their experience as seen as imperative for respondents to use the live stream ‘good’ and over a ﬁfth (23%) rated their experience as ‘excellent’, technology. Sense of responsibility means that the guidance Table 2. document that provides the method of assessment and supports the vital preparation required for the observation was shared in advance. Here respondents expressed the signiﬁcance of the ASSESSOR QUESTIONNAIRE FINDINGS guidance which they felt made a difference in the observation Expectation process. In response to the statement probing the assessors’ expectations One trainee reported, “I was given instructions and guidance about the effectiveness of assessing the trainees using the live from my tutor prior to my observations in order to prepare. My tutor stream observation, overall, assessors agreed that using the live also set time aside prior to the observation to ensure set up was ok, stream observation met their expectations (median = 3.5; IQR which also helped to ensure there was no disruption on the day.” 3.0–4.0; ρ = 0.74), Table 1. Two-third (67%) agreed that they In addition, a clinician reported, “I was guided by the trainees. effectively assessed the performance of the trainees. However, a They had done amazing well considering the situation. The girls have third (33%) were uncertain. For expectation based on overall needed very little support as they have been committed and worked satisfaction, the majority (87%) of assessors reported their hard on their own initiative.” experience as ‘good’, Table 3. Being organised. For the second theme, being organised, participants indicated that this was also important in preparing QUALITATIVE FINDINGS FROM TRAINEES’, CLINICIANS’, AND for the observation. For example, the usefulness of trainees being ASSESSORS’ QUESTIONNAIRES well organised before and during the livestream observation is Overall, 72 comments from trainees’, 49 comments from clinicians’ demonstrated by one clinician: and 15 comments from assessors were obtained in response to the open-ended questions. “Our trainee managed and organised herself outstandingly on this aspect. To be perfectly honest it was all sorted without any Theme Identiﬁcation input from myself. Obviously, my support was there should she The respondents (assessor, clinician, and trainee) were asked to need it. That goes for other members of the clinical team. Our enter qualitative comments on relevant areas covered in the trainee always explained to myself what her plans were and how questionnaire. Responses for clinicians and trainees were coded she carried them out.” into 5 domains (Preparation, Access and Usability, Equipment, Support and Feedback, and Expectation [Performance/Overall Satisfaction]) they were judged to best represent. Data corre- Inﬂuence of Time.Inﬂuence of Time as the third theme was sponding to more than one domain were coded accordingly. expressed as experiences of dichotomous relevance of time where Responses for assessors were related to effectively assessing using perceived support needs for the observation required an the live stream technology. Analysis of the data yielded distinct immediate response but once responded to, time became themes that represent the respondents’ experience of using the unimportant, with the respondents feeling well prepared for the live stream technology. observation. Trainees repeatedly expressed that they were supported with additional time to prepare for the observation. One trainee expressed this as “Sometimes it was hard to get observations within my workplace, but I got help around this when I needed to reach the deadlines.” Another trainee also touched on Table 3. Descriptive statistics for assessors. the step taken to obtain more time for the observation process: “Block the appointment for observations a bit longer than usual. It’s Measure/Questions Accessor helps me enough time to setting up and ﬁnishing up.” Expectation (effectiveness and overall satisfaction) Enabled me to effectively assess the performance of trainee dental nurses Team collaboration. The last theme related to preparation, team collaboration, was expressed as support that trainees received Agree n(%) 10 (66.7) from their team and the supporting clinicians, to prepare them for Uncertain n(%) 5 (33.3) the live stream observation process. Trainees commented on the Disagree n(%) – support from the administrative team who contacted patients and Overall satisfaction of your experience from colleagues who supported them by identifying ways to increase visibility during the observation. There was a general Good n(%) 13 (86.7) agreement for both trainees and clinicians that full support was Poor n(%) 1 (6.7) needed to ensure that the trainees were prepared, knew what was Undecided n(%) 1 (6.7) happening and what was expected of them: BDJ Open (2023) 9:4 C. Taylor et al. “My employer purchased a tripod to hold my phone during the A third participant responded that: “There were no issues with observations. Our reception team contacted the patient’s involved technology.” and explained the process.” Another trainee responded that, “We set up mock trials in each surgery to ﬁnd out where would be best to have the phone sitting.” Visual issues. The last theme, visual issues, was expressed as not being able to have clear visibility of all aspects normally assessed Clinicians expressed the importance of teamwork in providing within the observation process: support to the trainees: “I’m not sure how reliable a static stream of working area allows “We all work well as a team and feel the trainees were well for full observation and therefore full feedback in comparison to supported.” Also: “Trainee had good support from head nurse, an assessor being present in the room.” myself and from the associate they worked with. Between us we managed to ﬁnd what would work best in terms of positioning, However, some trainees highlighted how visibility issues were patient consent etc.” This suggests that team support is a key handled: element for ensuring effective preparation for and improved outcomes from the live stream observation. “During one of the observations my assessor didn’t have a clear view and made me aware of this which was sorted accordingly with the help of the clinician and assessor.” Themes for Access and usability Three distinct themes were identiﬁed that informed what access and usability meant to the participants: suitable alternative, Themes for Equipment technical constraints, and visual issues. One theme related to the need for additional equipment to hold and position the device for clearer visibility was identiﬁed: Suitable alternative. Suitable alternative was a theme expressing portability. that access and usability of the live stream technology worked well and live streaming is a ﬂexible and signiﬁcant tool that can be Portability. For the portability theme, most respondents indi- used to conduct workplace observation competency assessments. cated using a tripod as a platform for maintaining the stability of As one trainee said, their device during the live stream observation process. “Great tool used to observe skills and it may be something to “I used a large tripod which helped with the clear view of the consider to continue using in foreseeable future because it is surgery for my assessor.” convenient and efﬁcient.” However, respondents highlighted that other types of support, Similarly, clinicians also reported how useful it was for the such as ring lights, phone cases, lab work trolleys and phone trainees to continue their clinical practice during the pandemic. holders were used to support their devices during the process. “It was ﬁne and enabled things to continue despite the current “I have used a gooseneck phone holder. The phone holder is easy to secure on edge of table and it can be manipulated to what restrictions in clinical practice.” angle you want.” “[Trainee’s] observation went [well] she knew what was expected of her and the technology worked ﬁne.” Themes for Support and Feedback Five themes informing support and feedback experiences One of the positive aspects of using the live stream technology included process, visual support, educational value, technical was that whilst some respondents did report feeling anxious or support, and potential support if needed. stressed beforehand, when they began to use it, they reported feeling less anxious than they would have been in a face-to-face Process. Process was expressed as clear communication support scenario: provided by the assessors during the live stream observation. It was expressed by one trainee as: “I was very worried at the thought of the observation process when I started the course, but I think doing it via livestream “Because I was ﬁrst student for observations, I didn’t get the actually made it less stressful and nerve wracking.” document early enough before my observation, but I have been explained by my tutor verbally prior to my observation on the very same day.” Technical constraints. Technical constraints were described as the main challenge encountered and identiﬁed as a barrier Another clinician described this as an experience that left the to access and usability of the life stream technology. trainee fully supported during the observation and seems to Network issues, data usage and battery draining were express this theme well. regarded as problematic, especially as they were outside of the assessor’s control. However, some respondents highlighted “Overall, our trainee has been given the highest standard of their conﬁdence in and positive experiences in using the live support. Where our trainee struggled, there was always an stream: assessor there for guidance and support.” “Had no problems at any stage, felt conﬁdent all elements were able to be observed and assessed properly.” Another respondent Visual support. The theme of visual support was expressed by emphasised that: “I was nervous in case there was a drop in respondents as experiences of knowing when the assessor does signal which would interrupt observation but found that not have clear visibility. In this case, they can quickly inform the all assessments took place well without too many difﬁculties.” trainees about the visual support to enhance visibility. This theme BDJ Open (2023) 9:4 C. Taylor et al. encompassed several attributes including trainees’ and clinicians’ voiced concerns about technical problems such as network issues ability to know what to do when visibility is reduced. One trainee causing frequent disruption in the observation ﬂow; system issues shared her experience of increasing visibility during the observa- such as high-data usage and fast draining of battery; and tion process: infrastructural problems such as less visibility where assessors struggle to see clearly, and less value in using the live stream “During one of the observations my assessor didn’t have a clear technology appeared in the response for overall satisfaction. view and made me aware of this which was sorted accordingly Respondents thought that some of the challenges were com- with the help of the clinician and assessor.” pounded by the type of workplace building such as old buildings with thicker walls that restricted internet ﬂow. A trainee noted, “… When connection wasn’t very good it became Educational value. Educational value was a third theme in quite easy to get stressed to one ensure time keeping met while still support and feedback experiences and was described as trying to get good connection and also in case it got disconnected experiences of interpersonal interactions during the observation throughout.” process. These were characterised by perceptions of beneﬁcial Arranging speciﬁc patients for the observation due to busy support, good, helpful, and useful information, and feedback clinical lists was seen as a problem by clinicians, and the live towards the trainees and clinicians by the assessor. These observation was thought to be less useful. experiences were felt to enhance the individual’s competency. A clinician shared, “Due to Covid, live stream was the only way of This theme included attributes such as feeling comfortable and getting some form of observation, though there’s not much the conﬁdent, ﬂexibility of the assessor, being able to freely contact assessor can adequately see to assess from a single, ﬁxed vantage the assessor, and feeling good about the observation process point. It was difﬁcult to try to arrange speciﬁc patients in a row for because they were supported. One trainee highlighted how an the observations due to having a busy clinical list…I’m not assessors’ support led to a successful observation: convinced the live observation was useful in our nurse’s training.” “At the start of the observation the link wouldn’t allow me to join, Positive views for overall expectation. Some positive views of my assessor was able to support me in giving me more help to using the live stream technology included ease of using the live access the livestream.” stream technology; how less stressful it was in comparison to face- to-face assessment; how well trainees coped; the ability to Educational value was seen as signiﬁcant to support and complete the observation on time; how conﬁdent trainees felt in feedback in the live stream observation process, as expressed in being observed virtually; ease of contacting the assessor; and clear the words of one of the trainees: “I enjoyed the live stream visibility. The predominant positive view reported by both trainees observation because the feedback I received made me feel conﬁdent and clinicians was how the live stream technology worked well to in the fact that I was doing my job well and I will make a good ensure the assessment of trainees was performed during the dental nurse.” pandemic. One trainee shared: “It was good overall as was quite ﬂexible.” Technical support. Technical support as a theme represented Another trainee highlighted, “Live Streaming observation was difﬁculties experienced due to technological interferences of positive in the way that due to the current circumstances I was still internet connectivity and troubleshooting technology support able to be observed in my work, there was a couple of times due to working in a hospital setting it required slightly more planning but provided by the assessor. Respondents shared many experiences of how they were supported during technology problems and overall, I was happy I was able to continue with my course and do how personal examples of the assessor made a difference. One my observations due to the circumstances.” trainee shared, It was recommended that recording and compiling individual treatment would be much easier. Nevertheless, despite these “I had problem with connecting my integrated camera and my concerns other respondents also expressed that the use of the live tutor was telling me about her personal experience and how she stream technology was an appropriate alternative. This is shown in sorted out that problem. I was very stressed about it and helped some clinicians’ responses: me out a lot to calm down.” “Overall fantastic”. “I think the process of live stream assessment generally worked well in this scenario.” Potential support if needed. The last theme, potential support if needed, was expressed as being able to receive technological support if required but it was not needed. One clinician stated, ASSESSOR PERCEPTION FOR EFFECTIVELY ASSESSING “I personally did not require technology support; however, our PERFORMANCE USING THE LIVE STREAM TECHNOLOGY trainee was supported with any technical issues, and these were To effectively assess performance using live stream technology, resolved.” the factors which shaped the assessors’ perceptions have been investigated. In this case assessors were asked to comment on whether the live stream observations enabled them to effectively Themes for Expectation [Performance/Overall Satisfaction] assess the performance of trainee dental nurses. Assessors’ Two themes emerged for overall satisfaction with using the live perceptions of the effectiveness of using the live stream stream technology: negative and positive views for expectation. technology varied, and mixed responses were received. The Negative views mean respondents experienced some challenges ﬁndings have been grouped into two categories: factors aiding when using the live stream technology and were not satisﬁed. effectiveness and factors inhibiting effectiveness. Factors aiding Positive views, on the other hand mean that respondents effectiveness were efﬁciency, self-efﬁcacy, clear visibility, cost- experienced some valued beneﬁt when using the live stream effectiveness, and reduced travel time. Factors inhibiting effective- technology and were satisﬁed. ness were technical glitches (Wi-Fi connection, high data usage and sound quality), limited visibility, ﬁrst time use of digital Negative views for overall expectation. Some negative views of observation, and preferred method of observation (face-to-face/ using the live stream technology were identiﬁed. Respondents blended approach). BDJ Open (2023) 9:4 C. Taylor et al. Most assessors highlighted ‘some sort of [effective] strength’ during face-to-face observations); Achievements (Ability to associated with using the live stream technology. One assessor support trainees with better visibility during the observation said, “Most candidates coped well, and you were able to see enough process e.g., using phone holders that can hang to the wall; to note.” Another assessor expressed that he or she: “Was able to ability to complete qualiﬁcations during a pandemic and carry out effective observation of candidates performance during a upskilling of assessors by embracing this new environment to range of clinical dental procedures.” conduct the workplace assessment). The method used for the observation was felt to be all important. Another assessor emphasised on the tone of cost saving that: “There are a lot of beneﬁts no travel time or costs, if DISCUSSION patient cancels no time wasted for assessor, candidate can set up an This study contributes to knowledge in dental education as it observation last minute with assessor.” Being a suitable alternative reports on an innovative strategy that has the potential to change was also viewed as equally signiﬁcant: “Live stream observations the way in which workplace assessment is conducted. Consistent are certainly an obvious alternative to in-practice observations.” In with previous research into digital technologies for assessment addition, “Live stream observations have worked well. Initially there e.g. [1, 13–15, 22], we found that the beneﬁts of using live stream was some reluctance from practitioners and there has been slight technology for conducting the workplace observation assessment issue with sound at times but apart from that all good.” signiﬁcantly outweighed the challenges encountered. This use of However, several assessors questioned the effectiveness of digital technology strategy seems to enhance sustainable teach- using the live stream technology, discussing factors that inhibited ing, learning and assessment particularly during the COVID-19 its use. The difﬁculty of measuring effectiveness was raised with pandemic. The implementation of this innovative method has concerns of technical glitches, most assessors voiced concerns been successful and enabled assessment and progression of pre- regarding WIFI connection, data usage, sound quality, and limited registration dental nurses throughout the COVID-19 pandemic. visibility. One assessor highlighted, “For some yes, but others no, The successful implementation required assessors who created a very much depended on view and WIFI connection.” Nevertheless, consistent approach and supported the trainees to be familiar while some felt that there were some inhibiting factors of with the process. In addition, removing the requirement for an effectively assessing the trainees, others reserved criticism on the assessor to travel to visit a trainee’s workplace to conduct the basis of unprecedented circumstances that could be resolved. observation assessment and overall, signiﬁcantly reduced time taken to undertake this assessment process and cost of travel. “It can be difﬁcult to see everything and is connection dependant, Overall providing a more ﬂexible, sustainable approach to but I set up the meeting before the appointment time so that I arranging this method of assessment. To the best of our can observe the candidate setting up for the procedure and I ask knowledge, this is the ﬁrst study to evaluate the experiences of them to pan round the surgery. This also gives me the trainees’, clinicians, and assessors’ together on the use of live opportunity to ask any questions therefore fewer questions at stream observation. the end of the observation if the student does not have much The reliability of this study lay at an average of alpha value time.” greater than 0.6 across the instruments which is acceptable and there was a good response rate to completing the questionnaires. Another assessor expressed, “…What I was not able to capture, I The study represents a major change in workplace observation questioned my student after the procedure or arranged for a TEAMS assessment. The live stream observation was highly beneﬁcial for meeting when suitable for my candidate. Only once did I re-observe assessors as it decreased the need for in-practice assessment and the procedure.” efﬁciently compensated for the disruption of the trainees’ Among factors related to the effectiveness of using the live assessment. Overall, most trainees viewed the live stream stream technology, a preference for in person observation was a observation process positively, including preparing for the view held by some assessors, “I feel nothing can beat a face-to-face observation process, access and usability, feedback and support, observation, meeting with the student, the staff and looking at the and satisfaction. This is consistent with the ﬁndings from a study environment and building relations.” assessing undergraduate nursing students’ perspectives towards Another assessor emphasised, “It is very much depending on the video assessment of clinical skills that reported that students student, ability to get a good view of the working area and how accepted the video assessment process positively [14]. The much you can actually hear, especially regarding communication expectation component based on performance is an important with both the patient and the dentist. Still feel a blended approach is strength of using the live stream technology. a must to ﬁll in the blanks/make conﬁdent assessment decisions.” All the trainees indicated that the live stream observation enabled them to demonstrate their performance as dental nurses. The trainees were expected to demonstrate their capability for FOCUS GROUP FINDINGS FROM ASSESSORS safe and effective practice and maintain sufﬁcient ﬂexibility for We conducted three focus groups (n = 4, 4 and 5 participants, providing patient care while using live streaming. Based on our respectively) exploring assessors’ thoughts and experiences of ﬁndings, the live stream observation offered a safe working space, using the live stream observation with the following aims: ﬁrst, provided opportunity for the trainees to progress with their to identify potential information not already identiﬁed in the training, receive support from their assessors and feel conﬁdent survey; and second, to derive a meaningful assessment of user about their job. In agreement, most clinicians (82.9%) indicated experience in live stream observation as a group. The analysis of that the live stream observation enabled their trainees to the focus groups identiﬁed three core areas of reﬂections demonstrate their performance as a dental nurse. This contrasts relevant to using the live stream technology for workplace with the report of [15] who highlighted some concerns such as assessment: Beneﬁts (Convenience such as ability to organise nerves and performance being affected by using video assess- observations in limited time, ease of giving feedback, ﬂexibility ment but indicated its usefulness for assessing competency with an opportunity to build relationship with missed dental regards to feedback. However, our ﬁndings align with Kam and practices; Time effectiveness and cost effectiveness such as others (2019) who reported increased performance competence increased time for trainees to ask questions, reduced travel time of core basic nursing skill among participants that received video and possibility of short notice observation; reduced pressure); feedback [17]. In this study, the trainees also shared how they had Challenges (Technical difﬁculties such as poor internet connec- learnt from the personal experience of the assessors to remain tions, sound quality and less visibility; losing skills observed conﬁdent during the assessment process. BDJ Open (2023) 9:4 C. Taylor et al. The live stream observation therefore allowed the trainees to disciplines in the implementation of using technology to conduct present to the assessors how their practical experiences at the workplace assessment. The ﬁndings from this study and further practice had culminated into new insights, technological experi- analysis will inform where continual improvements can be made ences, and new learning. In addition, all the trainees reported their to effectively support students and workplace teams in the overall satisfaction experience with using the live stream was organisation, and facilitation of remote workplace observation good or excellent. Consequently, clinicians also expressed high competency-based assessment via live video stream. levels of overall satisfaction with using the live stream technology. Furthermore, (86.7%) of assessors reported that they had a good or very good overall satisfaction experience with the live stream CONCLUSIONS, RECOMMENDATIONS, AND FUTURE WORK observation. This is a signiﬁcant transformative outcome for Our main contribution in this paper is to investigate the views and technology-based workplace assessment in dentistry. Guze (2015) experiences of using live streaming for workplace observation highlighted the transformative learning experiences associated assessment to assess the overall satisfaction and effectiveness. with using technologies that engages the learner and Overall, it was found that satisfaction of using live streaming can be allows learning that does not endanger the patient [28]. It is linked with the resolving technology issues and support received, but also expected that assessors should be able to support effectiveness is mixed. Our results, as well as the limitations of learning into a more collaborative, personalised, and empowering supporting data, point naturally to several directions for future work. experience [28]. However, sustaining this method and the future use of remote live Across the three groups, median overall satisfaction rates were observation will be dependent on the Skills for Health decision to higher for trainees and clinicians than assessors (4.0, 4.0 and 3.0, maintain this as an assessment method with the assessment strategy respectively), which represents good to very good experience for the SVQ in Dental Nursing qualiﬁcation. Nevertheless, the use of among respondents. In addition, the median effectiveness score remote live observation assessed a range of generic and practical for assessors was 4.0, which shows that the assessors agreed with trainee competencies that require a higher level of conﬁdence in effectively assessing the performance of the trainees using the live managing the technology used and carrying on with the assessment. stream technology. These results suggest that trainees, clinicians, It allowed for feedback and validation of the trainee learning and assessors had a positive experience of using live stream experiences and was viewed as an efﬁcient and effective way of observation for the workplace observation assessment, and its use assessing and providing feedback particularly in response to global had an effective impact on trainee performance outcome and pandemics such as the changing condition posed by the ongoing achievement. Given the changing situation, as this method of COVID-19 pandemic. Thus, it is a feasible option for assessing trainees assessment is increasingly used, the assessors will likely better based in areas with transport difﬁculty (for example, practices in the understand its beneﬁt to improving trainees’ performance out- Western Isles) without having to travel. comes and reﬂective learning and their satisfaction rate will reﬂect NES has created a vision for Technology Enhanced Learning supporting learning effectiveness. that will enable education by facilitating appropriate, high-quality However, a key challenge in using live streaming is the technology Technology-Enhanced Learning for the Health and Social Care issue around internet stability which a concern for the trainees, workforce now and in the future. The ﬁndings and learning from clinicians, and assessors, as some were disconnected during the this project will be shared to support the integration and staff observation process. This may affect their abilities to be coordinated development of utilising the functionalities of various technolo- when performing the task during the observation process. Similarly, gies within blended learning programmes as a robust assessment using live streaming can constitute physical barriers for trainees who tool. Therefore, as the use of remote live observation continues to may be perplexed by their requirements, structure, and navigation. be developed, consideration should be given to incorporate Nevertheless, to combat this technology issues, before the observa- features that enable acceptance and re-imbursement for high- tion, a guidance document on this method of assessment and speed internet/phone expenses. Further local research and support for vital preparation was sent to the trainees and during the comparisons will help to identify practical and efﬁcient ways of assessment process, the assessors supported the trainees with a using live streaming to conduct workplace-based assessment and variety of troubleshooting techniques. The trainees, however, were give feedback to encourage learning among dental trainees. positive about using live stream observation as a tool for their Nevertheless, the use of the live stream technology generally workplace assessment stating that the guidance document provided worked well for the purpose of the workplace-based competency helped them prepare to use method for the assessment, allowed assessment. What was learned from the experience with the them to arrange a suitable time with their assessor and worked in a COVID-19 pandemic is the need for ﬂexibility concerning less stressful and nerve-wracking environment, even when visibility conducting the workplace-based assessment. was reduced. This is because the assessors made them aware they didn’t have a clear view and advised on best position to enhance REFERENCES visibility, which made the observation process run smoothly. 1. Rao S, Chowdhury N, Naithani M, Singh A, Singh N, Gupta AK. et al. Alternative strategy adopted in practical assessment for pathology postgraduate certiﬁcation examination amidst COVID-19 Nationwide lockdown: an Indian experience with a STUDY LIMITATIONS AND STRENGTH novel hybrid method of assessment. Adv. Med. Educ. Pract. 2020;12:1 https:// The limitation of the study is centred around the limited published www.ncbi.nlm.nih.gov/pmc/articles/PMC7802903/. data to support the use of live stream observation as a method of 2. El-Kishawi M, Khalaf K, Al-Najjar D, Seraj Z, Al Kawas S. Rethinking assessment workplace-based assessment. As this study involves a small concepts in dental education. Int. J. Dent.2020;2020:8672303 https:// number of dental nurse trainees, the ﬁndings may not be a www.ncbi.nlm.nih.gov/pmc/articles/PMC7584943/pdf/IJD2020-8672303.pdf. reﬂective of the personal experiences and viewpoints of all dental 3. Williams JC, Baillie S, Rhind SM, Warman S, Sandy J, Ireland A. A Guide to nurse trainees, clinicians, and assessors of other training providers. Assessment in Dental Education. 2015. https://research-information.bris.ac.uk/en/ publications/a-guide-to-assessment-in-dental-education. However, a key strength of the study is that it investigated views 4. Grieveson B, Kirton JA, Palmer NOA, Balmer MC. Evaluation of workplace based and experiences of using live stream observation from different assessment tools in dental foundation training. Br Dent J.2011;211:E8–E8. https:// perspectives and took a mixed methods approach to validate the www.nature.com/articles/sj.bdj.2011.681.pdf. ﬁndings. Moreover, this research indicates that this method 5. Kalsi HK, Kalsi JS, Fisher NL. An explanation of workplace-based assessments in was effective, and support from NES’s DCP workstream team postgraduate dental training and a review of the current literature. Br Dent signiﬁcantly contributed to the successful implementation. It is J.2013;215:519–24. https://www.nature.com/articles/sj.bdj.2013.1098.pdf. hoped that this paper can support assessors in dentistry and other BDJ Open (2023) 9:4 C. Taylor et al. 6. Manekar VS, Radke SA. Workplace based assessment (WPBA) in dental education- 28. Guze PA. Using technology to meet the challenges of medical education. Trans. A review. J Educ Technol Health Sci.2018;5:80–85. https://www.ipinnovative.com/ Am. Clin. climatological Assoc.2015;126:260. https://www.ncbi.nlm.nih.gov/pmc/ journal-article-ﬁle/7503. articles/PMC4530721/pdf/tacca1260000260.pdf. 7. Uma E. Workplace-based assessment: A valuable tool in undergraduate dental education. Int. J. Appl. Basic Med. Res.2020;10:223. https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC7758789/. ACKNOWLEDGEMENTS 8. General Dental Council. Employment and Registration. https://www.gdc-uk.org/ The authors would like to express their gratitude to all respondents who participated information-standards-guidance/supporting-the-dental-team/employment-and- in the study. We are also thankful and appreciate the help and support provided by registration#employment-and-roles-and-responsibilities-of-student-dental- the following staff: Pam Nicoll, Programme Director RRHEAL, NHS Education for nurses-and-technicians. Scotland. Alan Gillies, Knowledge Manager, NHS Education for Scotland. Dental Care 9. Skills for Health. Supplementary guidance for SVQ Dental Nursing approved centres to Professionals Workstream, NHS Education for Scotland. Dental Clinical Effectiveness support learners impacted by the COVID-19 pandemic. 2020. https://www.sqa.org.uk/ Workstream, NHS Education for Scotland sqa/ﬁles_ccc/svq-dental-nursing-supplementary-guidance-covid19.pdf. 10. Scottish Qualiﬁcations Authority. Scottish Vocational Qualiﬁcation in Dental Nursing. Available at SVQ Dental Nursing—SQA. https://www.sqa.org.uk/sqa/67607.html. AUTHOR CONTRIBUTIONS 11. Scottish Qualiﬁcations Authority. Guidance on assessment of Scottish Voca- CT conceived the study, acquisition of the data and initial draft preparation. AI tional Qualiﬁcations (SVQs). 2013. https://www.sqa.org.uk/ﬁles_ccc/ developed the methods and strategies of the data analysis, performed the data DB6083_Guidance_on_Assessment_of_SVQs.pdf. analysis, data interpretation and wrote the original draft manuscript. LY checked the 12. Brimble M. Skills assessment using video analysis in a simulated environment: an data analysis, critically revised the manuscript, and provided ﬁnal approval of the evaluation. Nursing Children and Young People. 2008;20. https://doi.org/10.7748/ manuscript. All authors reviewed and edited the ﬁnal draft manuscript. All authors paed2008.09.20.7.26.c6705. have read and agreed to the published version of the manuscript. 13. Forbes H, Oprescu FI, Downer T, Phillips NM, McTier L, Lord B. et al. Use of videos to support teaching and learning of clinical skills in nursing education. A review. Nurse Education Today 42, 53–56. Harden, R., 1988. What is an OSCE?. Med. Teach.2016;10:19–22. https://pubmed.ncbi.nlm.nih.gov/27237353/. COMPETING INTERESTS 14. Lewis P, Hunt L, Ramjan LM, Daly M, O’Reilly R, Salamonson Y. Factors con- The authors declare that they have no competing interests with respect to the tributing to undergraduate nursing students’ satisfaction with a video assess- authorship and/or publication of this article. ment of clinical skills. Nurse Educ. today. 2020;84:104244. https:// pubmed.ncbi.nlm.nih.gov/31715471/. 15. Ho RC, Song BL. Immersive live streaming experience in satisfying the learners’ ETHICS APPROVAL need for self-directed learning. Interact. Technol. Smart Educ. 2021; 10.1108/ITSE- The study was assessed using the NHS Health Research Authority deﬁning research 12-2020-0242. online decision-making tool. This tool conﬁrmed that the study would not be considered 16. Snekalatha S, Marzuk SM, Meshram SA, Maheswari KU, Sugapriya G, Sivasharan K. research by the NHS and that NHS Research Ethics Committee review was not required. Medical students’ perception of the reliability, usefulness and feasibility of The study was governed in accordance with NES’s guidance for the conduct of unproctored online formative assessment tests. Adv Physiol Educ.2021;45:84–88. evaluation studies, and institutional ethical review was also not required. https://doi.org/10.1152/advan.00178.2020. 17. Kam BS, Yune SJ, Lee SY, Im SJ, Baek SY. Impact of video feedback system on medical students’ perception of their clinical performance assessment. BMC Med INFORMED CONSENT Educ.2019;19:1–7. https://doi.org/10.1186/s12909-019-1688-6. Informed consent was obtained from all participants involved in the process: trainees, 18. Weeks DL, Molsberry DM. Feasibility and reliability of remote assessment of PALS clinicians, and assessors. The data used for the study was pseudonymised before the psychomotor skills via interactive videoconferencing. Resuscitation. 2009;80:354–8. analysis was conducted and managed according to NES privacy policy. https://www.sciencedirect.com/science/article/abs/pii/S0300957208007831. 19. Choy I, Fecso A, Kwong J, Jackson T, Okrainec A. Remote evaluation of laparo- scopic performance using the global operative assessment of laparoscopic skills. ADDITIONAL INFORMATION Surg Endosc.2013;27:378–83. https://pubmed.ncbi.nlm.nih.gov/22890477/. 20. McQueen S, McKinnon V, VanderBeek L, McCarthy C, Sonnadara R. Video-based Correspondence and requests for materials should be addressed to Caroline Taylor. assessment in surgical education: a scoping review. J. Surg Educ. 2019;76:1645–54. https://pubmed.ncbi.nlm.nih.gov/31175065/. Reprints and permission information is available at http://www.nature.com/ 21. Wang Y, Dore K, Wong N, Hoogenes J, Matsumoto E. MP34-12 Utility and feasi- reprints bility of video-based assessment of residents performing ureteroscopy and laser lithotripsy. J. Urol. 2020;203:e507–e508. https://www.auajournals.org/doi/abs/ Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims 10.1097/01.JU.0000556009.17548.ba. in published maps and institutional afﬁliations. 22. Angelo RL, Pierre PS, Tauro J, Gallagher AG. Live Observational Objective Assessment of Operative Performance in a Cadaveric Model is Equivalent to Delayed Video-Based Assessment. Arthrosc.: J. Arthroscopic Relat. Surg.2021;37:3241–7. https://pubmed.ncbi.nlm.nih.gov/33964394/. Open Access This article is licensed under a Creative Commons 23. Pugh CM, Hashimoto DA, Korndorffer JR,Jr. The what? How? And who? Of video Attribution 4.0 International License, which permits use, sharing, based assessment. Am. J. Surg.2021;221:13–18. https://pubmed.ncbi.nlm.nih.gov/ adaptation, distribution and reproduction in any medium or format, as long as you give 32665080/. appropriate credit to the original author(s) and the source, provide a link to the Creative 24. IBM Corp. IBM statistical package for social sciences (SPSS) for windows, (version Commons license, and indicate if changes were made. The images or other third party 27.0). Armonk, NY, USA: IBM Corp; 2020. material in this article are included in the article’s Creative Commons license, unless 25. Dall’Oglio AM, Rossiello B, Coletti MF, Caselli MC, Ravà L, Di Ciommo V. et al. indicated otherwise in a credit line to the material. If material is not included in the Developmental evaluation at age 4: Validity of an Italian parental ques- article’s Creative Commons license and your intended use is not permitted by statutory tionnaire. J. Paediatr Child Health. 2010;46:419–26. https://doi.org/10.1111/ regulation or exceeds the permitted use, you will need to obtain permission directly j.1440-1754.2010.01748.x. from the copyright holder. To view a copy of this license, visit http:// 26. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative creativecommons.org/licenses/by/4.0/. Res. Psychol.2006;3:77–101. https://www.tandfonline.com/doi/abs/10.1191/ 1478088706qp063oa. © The Author(s) 2023 27. Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: Striving to meet the trustworthiness criteria. Int. J. qualitative methods. 2017;16:1609406917733847 https://journals.sagepub.com/doi/full/10.1177/1609406917733847. BDJ Open (2023) 9:4 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png BDJ Open Springer Journals http://www.deepdyve.com/lp/springer-journals/using-live-stream-technology-to-conduct-workplace-observation-PQ4E64Qd0d

Loading next page...

Publisher: Springer Journals
Copyright: Copyright © The Author(s) 2023
eISSN: 2056-807X
DOI: 10.1038/s41405-023-00132-0
Publisher site: See Article on Publisher Site

Abstract

www.nature.com/bdjopen ARTICLE OPEN Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience 1✉ 2 1 3 1 1 1 Caroline Taylor , Adalia Ikiroma , Anne Crowe , David H Felix , Gillian Grant , Lucy Mitchell , Teresa Ross , 1 2 Margaret Saunderson and Linda Young © The Author(s) 2023 AIM/OBJECTIVE: This study evaluates the effectiveness and users’ experience of using live stream technology to conduct workplace observation assessments of trainee dental nurses. Information on the usability, accessibility, and general satisfaction of this technological technique were collected. MATERIALS AND METHODS: This nationwide cross-sectional survey was conducted in Scotland and included one focus group and three online questionnaires with qualitative and quantitative questions. The quantitative responses were described using standard descriptive analysis, while the quantitative data were investigated using thematic analysis. RESULTS: Eighty-one trainee dental nurses, 35 clinicians and 19 assessors participated in this study. Live stream observation was generally well received by the trainee dental nurses and clinicians, who thought that it had helped increase their conﬁdence to perform practical skills. The assessors also stated that overall satisfaction was high, and that live stream observation met their expectations for efﬁcacy. However, several technical challenges, such as network issues were brought up by responders. CONCLUSION: This study provides evidence that workplace observation assessments can be performed in the future by using live stream technology. However, additional investigation and comparison will aid in determining the most effective way of using this approach and providing feedback to promote learning among dental trainees. BDJ Open (2023) 9:4 ; https://doi.org/10.1038/s41405-023-00132-0 INTRODUCTION Dental Nurses are members of a group of Dental Care The coronavirus disease-19 (COVID-19) pandemic resulted in Professionals (DCPs) who play a vital role within a dental team global lockdown and created disruptions in educational delivery and provide support and assistance in clinical and non-clinical and assessment processes. Many educational activities including aspects in providing high-quality patient care. All dental profes- course completion examinations and certiﬁcation were either sionals must be registered with the General Dental Council (GDC) postponed or cancelled. Of particular concern was the delivery of before they can work in the United Kingdom [8]. It is a robust and timely observational assessment of practical skills [1]. requirement that dental nurses undertake a pre-registration Assessment is regarded as a crucial component in the learning training programme with a training provider approved by the process [2], and it should be sustainable if it is to foster lifelong GDC towards GDC recognised qualiﬁcation. Mindful of this, Skills learning [3]. This suggests that no matter what assessment for Health recognised that assessors may ﬁnd it more difﬁcult to method is utilised, it should effectively stimulate engagement to gain access to a dental environment to carry out observations improve learning and performance, and thus promote sustainable during COVID-19 restrictions and authorised the use of live video development of learning capabilities. Emphasis on workpla- stream as a suitable assessment method [9]. Skills for Health is a ce‑based assessment (WBA) in dental health professionals’ training non-proﬁt organisation committed to the development of an has been increasing in the last decade [4–7]. Most WBAs are improved and sustainable healthcare workforce across the United observational and help the assessor to assess a trainee’s Kingdom. professional competence in a range of clinical scenarios. They Observation of practice is a primary source of evidence and a also provide the opportunity for timely and objective feedback to fundamental principle of the Scottish Vocational Qualiﬁcation the learner about both their clinical and non-clinical soft skills [7]. (SVQ) in Dental Nursing qualiﬁcation [10]. The SVQ standards A key aim of feedback is to encourage and support learners to require assessment methods to be selected on the basis of their self-assess and reﬂect on their learning to help them self-correct internal veriﬁcation process, which is made up of its validity, and orient their current performance. reliability, and practicability [11]. The validity of an assessment is 1 2 Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK. Dental Clinical Effectiveness Workstream, NHS Education for Scotland, Edinburgh, UK. Dental Directorate, NHS Education for Scotland, Edinburgh, UK. email: Caroline.taylor@nhs.scot Received: 8 November 2022 Revised: 22 December 2022 Accepted: 13 January 2023 1234567890();,: C. Taylor et al. the degree to which the assessment method is appropriate to the METHODS standards. The reliability of an assessment is the degree of Design This was a national cross-sectional survey conducted in Scotland consistent results when used with different candidates, different comprising three online questionnaire and one focus group. assessors and on different occasions. The practicability of an assessment refers to the feasibility that the assessment makes best use of available resources, equipment, and time. Prior to the Participants COVID-19 pandemic, in order to conduct a WBA all assessors were The study participants included all trainee dental nurses undertaking pre- registration training with NES, the clinicians they provided chairside required to travel to the trainee’s workplace. Depending on the support to and assessors who completed the live stream observation geographic distance from the assessor’s base to the trainee’s assessment process. The clinicians were dentists, dental therapists, and workplace, conducting the observation might result in at least two hygienists who work with the trainees in their practice, and the assessors visits which made a signiﬁcant impact on travelling time were Dental Tutors employed by NES, who carry out the role of an and costs. Assessor. El-Kishawi et al. highlighted that evolving technologies and methods of assessment could enhance students’ learning environ- Description of the Assessment Process ments and improve tutors’ assessment experiences [2]. Some studies Evidence of candidates’ performance is drawn primarily from naturally have reported students’ positive learning experience from using occurring work activities that take place under normal working conditions digital technology for assessment in both medical and non-health in a normal work environment. This WBA requires an assessor to observe a disciplines [1, 12–16]. These studies have uncovered beneﬁts at the trainee dental nurse while carrying out practical based tasks as deﬁned in tertiary education level such as increased access and ﬂexibility [1], the qualiﬁcation standards as they provide chairside support to a GDC the provision of a platform for an interactive and independent registered dentist, hygienist, or therapist providing care to the patient. This learning opportunity [15], a useful tool for improving practice and is referred to as a Performance Criteria, and it describes the level of competency required for the SVQ in Dental Nursing qualiﬁcation. This assessing competency [12], and the opportunity to develop skills in observation is conducted in the trainee’s own workplace, which could be a the use of computer technology [13]. Additional beneﬁts beyond General Dental Practice, Public or Hospital Dental Service. Students skill acquisition were identiﬁed in a recent study which found that pursuing the SVQ in Dental Nursing must be observed supporting the video feedback promoted greater knowledge retention, identiﬁed patient and the clinician as they provide chairside support during a variety performance situation, clinical reasoning, self-efﬁcacy, and learning of clinical dental procedures and duties within their role. The assessor satisfaction experience among students [17]. The outcome from records their performance and provides feedback on their ﬁndings. This video feedback was greater compared to computer assisted provides evidence for the trainee’s portfolio to demonstrate achievement. assessment analysis report or online-written comment. Assessors must be competent in the area of practice to which the speciﬁc To ensure uninterrupted progress and reduce the impact of the national occupational standards apply, according to the Scottish Qualiﬁca- tions Authority (SQA), and for the SVQ in Dental Nursing, assessors must COVID-19 pandemic restrictions on trainee dental nurses, NHS hold a qualiﬁcation recognised by the regulatory body for enrolment and Education for Scotland’s (NES) DCP Workstream devised the process maintain professional registration with the GDC as well as obtain an to align the Skills For Health COVID-19 supplementary guidelines for assessor qualiﬁcation. the assessment strategy. With reports that the use of real-time Microsoft Teams Platform (https://www.microsoft.com), a workspace for online assessment has good prospects [1] and culminates in positive real-time collaboration and communication was used during the assess- academic outcomes [18], a process utilising live video stream to ment. Microsoft Teams is freely available for download, with secure conduct observational WBAs and effectively assess performance was connection, and features that allow multiple participants and screen designed and applied. Research exploring the use of digital sharing. The meeting link was shared with all participants, allowing them technology has investigated the reliability and accuracy of live to connect, the assessor to observe, and interact during the practical observation. Before the observation, a test run was conducted to ensure stream observation for performance assessment and feedback; connectivity and video/audio quality. [1, 19–23] however, research evaluating the use of live stream technology at workplace level is in its infancy. Wang et al. reported its usefulness and feasibility with high interrater reliability of video- Questionnaire Development based assessment of medical residents’ performance [21]. McQueen There were three questionnaires developed for data collection: one for dental nurse trainees, one for clinicians, and one for assessors. The majority et al. highlighted the beneﬁts of video-based assessment including of the sections in the trainee and clinician questionnaires were the same. the ability to capture clinical ability in the operating room without The trainee questionnaire was divided into ﬁve sections to collect data on decreasing intraoperative efﬁciency and the potential to improve (1) preparation (seven questions), (2) access and usability (four questions), formative assessment and feedback practices [20]. Similarly, Rao (3) equipment (two questions), (4) support and feedback (four questions et al. found that 85% of participants strongly agreed and 15% for clinicians and ﬁve questions for trainees), and (5) expectation agreed that essential minor deviations during real-time online [performance/overall satisfaction] (two questions). Both questionnaires practical assessment did not compromise the basic principles and included open-ended questions for comments on the aforementioned goals of assessment, and the quality of assessment through this measures. The assessor questionnaire included 3-items to collect informa- mode of assessment was graded as good to excellent [1]. Therefore, tion about expectations: [effectiveness/overall satisfaction] (two questions) and an open-ended question about effectively assessing trainee the evidence suggests digital technology can transform the way performance. workplace‑based assessment is delivered. Some of the closed-ended question items required a binary response However, very little is known about the use of technology in (Yes/No). Most questions were based on a 3-point Likert scale with ‘3’ performing workplace assessment in dental education. It is indicating never and ‘1’, all the time to statements in the access and important to consider whether it can be objectively measured usability domain; a 4-point Likert scale with ‘4’ indicating immediately and and evaluate users’ perceptions of its usefulness and effectiveness, ‘1’, several days later to statements in the feedback domain; and a 5-point particularly in the post-pandemic era. This knowledge is key to Likert scale with ‘5’ indicating strong agreement and ‘1’, strong integrating and sustaining the use of live stream technology in disagreement to statements in the preparation, support and feedback, vocational dental education programmes which are actively performance, effectiveness, and overall satisfaction domains, respectively. seeking opportunities to embed technology-based education to improve their programme. Therefore, the focus of this study was Data Collection to explore the views of trainee dental nurses, clinicians, and This was an online questionnaire survey hosted on Questback which is the assessors to evaluate users’ experiences and satisfaction of using NES approved online survey platform. This was distributed by email live stream technology to conduct workplace observational containing a secure link to the survey to the trainees and clinicians after completing the live stream observation assessment, whilst the assessors assessments. BDJ Open (2023) 9:4 C. Taylor et al. before the observation. Ninety-six percent agreed that they Table 1. Median rating and reliability according to the measures for followed the guidance for the observation, (93%) agreed that respondents. the guidance was discussed with their teams and (87%) agreed that their workplace provided support in preparation for the Category Measure x͂ IQR α/ρ observation. Eighty-seven percent agreed there were able to test Trainee 0.72 the process with an assessor before the observation and (89%) Preparation 5.00 4.00–5.00 0.63 agreed that their workplace provided support in preparing Access and usability 3.00 2.00–3.00 0.63 them for the observation. In addition, (96%) of the respondents Feedback and support 5.00 4.00–5.00 0.61 agreed that patients were comfortable with the method used for the observation, Table 2. Expectation 4.50 4.00–5.00 0.69 Clinicians agreed that their trainees were prepared for the live (performance and stream observation (median = 4.0 IQR 3.0-4.0; α = 0.65), Table 1. overall satisfaction) The majority (91%) also received the guidance document that Clinician 0.82 provided information on the method of assessment and vital Preparation 4.00 3.00–4.00 0.65 support for the trainees before the observation. Over three- Access and usability 3.00 2.00–3.00 0.80 quarters (77%) agreed that the instruction within the guidance was clear to follow. Eighty-three percent agreed that they Feedback and support 3.00 3.00–4.00 0.68 discussed the content of the guidance document with their Expectation 4.00 3.50–4.50 0.84 trainee in advance of the observation. In addition, 71% agreed (performance and that they were able to test the process with the assessor before overall satisfaction) the observation and 94% agreed that patients involved in the live Assessor Expectation 3.50 3.00–4.00 0.81 stream observation were comfortable with the method used. (effectiveness and Eighty-seven percent agreed that they felt comfortable with using overall satisfaction) technology for the trainee’s observations. The majority (91%) of Note. x-͂ Median; IQR- Inter-Quartile Range α/ρ-Cronbach’s alpha/Spearman- clinicians reported that overall, they felt that sufﬁcient support Brown coefﬁcient was received from the assessor in preparing for the trainee’s The value for equipment domain was not included in the table as the internal observation, Table 2. consistency was not satisfactory Access and usability Trainees agreed that the live stream observation was accessible completed their questionnaires at the end of the observation assessment period. Data were collected from November 2020 for a period of two and easy to use (median = 3.0; IQR 2.5–3.0; ρ = 0.63). Just under weeks to complete after the live stream observation assessment. In three quarters (73%) reported that they never had difﬁculties addition, invitations to participate in online focus groups via Microsoft accessing suitable equipment to undertake the observation. Most Teams were circulated to all assessors employed by NES. (89%) reported that they were able to access the live stream with ease all the time to undertake the observation. In addition, more Data Analysis than half of the trainees (64%) reported that they had reliable Standard descriptive analyses were performed to describe the responses connectivity all the time throughout the observation, and (69%) using the Statistical Package for the Social Sciences (SPSS) version 27 [24]. achieved clear view of the assessor to review their performance all For each, median (and interquartile range [IQR]) rating was used as central the time, Table 2. tendency (central quartile range) and internal consistency using Cron- Clinicians agreed that the live stream observation was accessible bach’s alpha or Spearman-Brown coefﬁcient. A cut-off value of 0.6 was and easy to use (median = 3.0; IQR 2.0–3.0; α= 0.80), Table 1.Around considered a good internal consistency of the items in the scale [25]. Items two-thirds of clinicians (66%) reported that they never had difﬁculties were reverse scored when necessary; therefore, the higher the score, the accessing suitable equipment to undertake the observation. More more positive the response. The open-ended data were explored using than half (54%) reported that they were able to access the live stream thematic analysis [26, 27] to identify themes which emerged from the data with ease all the time to undertake the observation. About half (51%) within the measures. This involved organising the qualitative data into themes and was performed on Microsoft Excel. reported that they had reliable connectivity all the time throughout the observation, and around two-thirds (66%) achieved a clear view of the clinical area for the assessor to view their trainee’s Results performance all the time, Table 2. Out of 81 trainees that participated in the live stream observation, 45 completed the questionnaire with a response rate of 55.6%. Additionally,19 assessors participated in the live stream observation, although only 15 of Equipment them, together with 35 clinicians, completed the questionnaire. The For the equipment measure, the Spearman-Brown coefﬁcient for response rate was 78.9% for assessors; however, it was not possible to both trainees (ρ = 0.36) and clinicians (ρ = 0.48) was <0.6, and calculate this for the clinicians as the total number involved in the trainee’s therefore an overall measure was not calculated. For the type of IT remote observation assessment is unknown. In line with the research equipment used for the observation, more than half of the question, the collected data provided information about the participants’ trainees (62%) and around two-third (66%) of clinicians used smart perspectives on the effectiveness and overall satisfaction of using live phone while (16%) and (11%) used laptop. In addition, more than stream technology to conduct workplace observation competency half (62%) of the trainees and around two-third (66%) of clinicians assessment of trainee dental nurses. The internal consistency coefﬁcient across all items for trainees, clinicians and assessors were 0.72, 0.82, and reported that they did not use additional equipment to hold or 0.81, respectively, Table 1. position their device to undertake the observation, while over a third 38% and just over a third (34%) used one to hold or position their device, Table 2. TRAINEE AND CLINICIAN QUESTIONNAIRE FINDINGS Preparation Feedback and support Trainees agreed that they were prepared for the live stream Overall, trainees agreed that they were supported for the live observation (median = 5.0; IQR 4.0-5.0; α = 0.63), Table 1. The stream observation (median = 5.0; IQR 4.0–5.0; α = 0.61), Table 1. majority (93%) received the guidance document that provided Roughly half, 47% of respondents agreed that the assessor information on the method of assessment and vital support provided technology/IT assistance when needed, 98% agreed that BDJ Open (2023) 9:4 C. Taylor et al. Table 2. Descriptive statistics for trainees and clinicians. Table 2. continued Measure/Questions Trainees Clinicians Measure/Questions Trainees Clinicians Expectation (performance and overall satisfaction) The feedback received: Was helpful in supporting development Allowed demonstration of performance as a dental nurse Agree n(%) 45 (100) 29 (82.9) Agree n(%) 45 (100) 29 (82.9) Uncertain n(%) – 5 (14.3) Uncertain n(%) – 5 (14.3) Disagree n(%) – 1 (2.9) Disagree n(%) – 1 (2.9) Provide any technology/IT assistance when needed Overall satisfaction of your experience Yes n(%) 21 (46.7) 14 (40.0) Excellent n(%) 20 (44.4) 8 (22.9) No n(%) 24 (53.3) 21 (60.0) Good n(%) 25 (55.6) 26 (74.3) Access and usability Poor n(%) – 1(2.9) Difﬁculties accessing suitable equipment needed for the observation Preparation All of the time n(%) 1 (2.2) 1 (2.9) Received the live stream observation guidance document Some of the time n(%) 11 (24.4) 11 (31.4) Yes n(%) 42 (93.3) 32 (91.4) Never n(%) 33 (73.3) 23 (65.7) No n(%) 3 (6.7) 3 (8.6) Able to access the live stream session with ease The instructions within the guidance was clear to follow All of the time n(%) 40 (88.9) 19 (54.3) Agree n(%) 43 (95.6) 27 (77.1) Some of the time n(%) 5 (11.1) 13 (37.1) Uncertain n(%) 1 (2.2) 7 (20.0) Never n(%) – 3 (8.6) Disagree n(%) 1 (2.2) 1 (2.9) Connectivity reliable throughout the observation Discussed the content of the guidance document All of the time n(%) 29 (64.4) 18 (51.4) Agree n(%) 42 (93.3) 29 (82.9) Some of the time n(%) 15 (33.3) 16 (45.7) Uncertain n(%) 2 (4.4) 5 (14.3) Never n(%) 1 (2.2) 1 (2.9) Disagree n(%) 1 (2.2) 1 (2.9) Able to achieve a clear view of the clinical area Able to test the process before the observation All of the time n(%) 31 (68.9) 23 (65.7) Agree n(%) 39 (86.7) 25 (71.4) Some of the time n(%) 14 (31.1) 9 (25.7) Uncertain n(%) – 6 (17.1) Never n(%) – 3 (8.6) Disagree n(%) 6 (13.3) 4 (11.4) Equipment Workplace provided support in preparing for the observation Type of IT equipment used Agree n(%) 40 (88.9) 32 (91.4) Android 3 (6.7) 1 (2.9) Uncertain n(%) 4 (8.9) – Desktop/PC 2 (4.4) 4 (11.4) Disagree n(%) 1 (2.2) 3 (8.6) Laptop 7 (15.6) 4 (11.4) Patients involved were comfortable with the method used Tablet 5 (11.1) 3 (8.6) Agree n(%) 43 (95.6) 33 (94.3) Smart phone 28 (62.2) 23 (65.7) Uncertain n(%) 2 (4.4) 2 (5.7) Additional equipment to hold/position device Disagree n(%) –– Yes n(%) 17 (37.8) 12 (34.3) Felt comfortable using technology for the workplace observation No n(%) 28 (62.2) 23 (65.7) Agree n(%) 43 (95.6) 31 (88.6) Different question for clinicians—Overall, did you feel that you received sufﬁcient support from the assessor in preparing for the trainee’s Uncertain n(%) 2 (4.4) 2 (5.7) observation? Disagree n(%) – 2 (5.7) Trainee: The feedback received was: Timely; Clinician: Did the trainee share their assessor feedback with you? Feedback and support How soon was feedback received after the observation the feedback from the assessor was timely, all respondents (100%) Immediately n(%) 27 (60.0) – agreed that the feedback from the assessor was clear and Same day n(%) 14 (31.1) – constructive and supported their development, Table 2. Next day n(%) 3 (6.7) – Overall, clinicians’ perspective towards feedback and support, Several days later n(%) 1 (2.2) – was good, (median = 3.0; IQR 3.0–4.0; α = 0.68), Table 1. There was an equal agreement (83%) among clinicians with the statement The feedback received: Timely / Shared the assessor feedback “feedback received was clear and constructive” and “feedback Agree n(%) 44 (97.8) 30 (85.7) received was helpful in supporting trainee development”. The vast Uncertain n(%) 1 (2.2) – majority (86%) of clinicians said the trainees shared their assessor Disagree n(%) – 5 (14.3) feedback with them. However, when asked if the assessor provided technology/IT assistance when needed, more than half The feedback received: Was clear and constructive (60%) of clinicians said ‘no’, Table 2. Agree n(%) 45 (100) 29 (82.9) Uncertain n(%) – 6 (17.1) Expectation Disagree n(%) –– Overall, trainees agreed that using the live stream observation met their expectations (median = 4.5; IQR 4.0–5.0; ρ = 0.69), BDJ Open (2023) 9:4 C. Taylor et al. Table 1. To evaluate trainees’ expectation based on perfor- CLINICIAN AND TRAINEE mance, all respondents (100%) agreed that the live stream Themes for Preparation observation allowed them to demonstrate their performance. Both trainees and clinicians reported similar themes that For expectation based on overall satisfaction, nearly one-half enhanced their preparation for the live stream observation. Four (44%) reported their experience as ‘excellent’ and 56% had a distinct themes were identiﬁed that describe what it meant to ‘good’ experience from using the live stream observation for participants to be prepared for the observation process. These their assessment, Table 2. four themes are sense of responsibility, being organised, inﬂuence of Overall, clinicians agreed that using the live stream observa- time, and team collaboration. The participants described how tion met their expectations (median = 4.0; IQR 3.5–4.5; ρ = 0.84), important these themes were with regards to the type of support Table 1. To evaluate clinicians’ expectation based on perfor- provided by their team and their preparedness for the mance, The majority (83%) of clinicians agreed that the live observation. stream observation allowed their trainees to demonstrate their performance. For expectation based on overall satisfaction, just Sense of responsibility. The ﬁrst theme, sense of responsibility was under three quarters (74%) of clinicians rated their experience as seen as imperative for respondents to use the live stream ‘good’ and over a ﬁfth (23%) rated their experience as ‘excellent’, technology. Sense of responsibility means that the guidance Table 2. document that provides the method of assessment and supports the vital preparation required for the observation was shared in advance. Here respondents expressed the signiﬁcance of the ASSESSOR QUESTIONNAIRE FINDINGS guidance which they felt made a difference in the observation Expectation process. In response to the statement probing the assessors’ expectations One trainee reported, “I was given instructions and guidance about the effectiveness of assessing the trainees using the live from my tutor prior to my observations in order to prepare. My tutor stream observation, overall, assessors agreed that using the live also set time aside prior to the observation to ensure set up was ok, stream observation met their expectations (median = 3.5; IQR which also helped to ensure there was no disruption on the day.” 3.0–4.0; ρ = 0.74), Table 1. Two-third (67%) agreed that they In addition, a clinician reported, “I was guided by the trainees. effectively assessed the performance of the trainees. However, a They had done amazing well considering the situation. The girls have third (33%) were uncertain. For expectation based on overall needed very little support as they have been committed and worked satisfaction, the majority (87%) of assessors reported their hard on their own initiative.” experience as ‘good’, Table 3. Being organised. For the second theme, being organised, participants indicated that this was also important in preparing QUALITATIVE FINDINGS FROM TRAINEES’, CLINICIANS’, AND for the observation. For example, the usefulness of trainees being ASSESSORS’ QUESTIONNAIRES well organised before and during the livestream observation is Overall, 72 comments from trainees’, 49 comments from clinicians’ demonstrated by one clinician: and 15 comments from assessors were obtained in response to the open-ended questions. “Our trainee managed and organised herself outstandingly on this aspect. To be perfectly honest it was all sorted without any Theme Identiﬁcation input from myself. Obviously, my support was there should she The respondents (assessor, clinician, and trainee) were asked to need it. That goes for other members of the clinical team. Our enter qualitative comments on relevant areas covered in the trainee always explained to myself what her plans were and how questionnaire. Responses for clinicians and trainees were coded she carried them out.” into 5 domains (Preparation, Access and Usability, Equipment, Support and Feedback, and Expectation [Performance/Overall Satisfaction]) they were judged to best represent. Data corre- Inﬂuence of Time.Inﬂuence of Time as the third theme was sponding to more than one domain were coded accordingly. expressed as experiences of dichotomous relevance of time where Responses for assessors were related to effectively assessing using perceived support needs for the observation required an the live stream technology. Analysis of the data yielded distinct immediate response but once responded to, time became themes that represent the respondents’ experience of using the unimportant, with the respondents feeling well prepared for the live stream technology. observation. Trainees repeatedly expressed that they were supported with additional time to prepare for the observation. One trainee expressed this as “Sometimes it was hard to get observations within my workplace, but I got help around this when I needed to reach the deadlines.” Another trainee also touched on Table 3. Descriptive statistics for assessors. the step taken to obtain more time for the observation process: “Block the appointment for observations a bit longer than usual. It’s Measure/Questions Accessor helps me enough time to setting up and ﬁnishing up.” Expectation (effectiveness and overall satisfaction) Enabled me to effectively assess the performance of trainee dental nurses Team collaboration. The last theme related to preparation, team collaboration, was expressed as support that trainees received Agree n(%) 10 (66.7) from their team and the supporting clinicians, to prepare them for Uncertain n(%) 5 (33.3) the live stream observation process. Trainees commented on the Disagree n(%) – support from the administrative team who contacted patients and Overall satisfaction of your experience from colleagues who supported them by identifying ways to increase visibility during the observation. There was a general Good n(%) 13 (86.7) agreement for both trainees and clinicians that full support was Poor n(%) 1 (6.7) needed to ensure that the trainees were prepared, knew what was Undecided n(%) 1 (6.7) happening and what was expected of them: BDJ Open (2023) 9:4 C. Taylor et al. “My employer purchased a tripod to hold my phone during the A third participant responded that: “There were no issues with observations. Our reception team contacted the patient’s involved technology.” and explained the process.” Another trainee responded that, “We set up mock trials in each surgery to ﬁnd out where would be best to have the phone sitting.” Visual issues. The last theme, visual issues, was expressed as not being able to have clear visibility of all aspects normally assessed Clinicians expressed the importance of teamwork in providing within the observation process: support to the trainees: “I’m not sure how reliable a static stream of working area allows “We all work well as a team and feel the trainees were well for full observation and therefore full feedback in comparison to supported.” Also: “Trainee had good support from head nurse, an assessor being present in the room.” myself and from the associate they worked with. Between us we managed to ﬁnd what would work best in terms of positioning, However, some trainees highlighted how visibility issues were patient consent etc.” This suggests that team support is a key handled: element for ensuring effective preparation for and improved outcomes from the live stream observation. “During one of the observations my assessor didn’t have a clear view and made me aware of this which was sorted accordingly with the help of the clinician and assessor.” Themes for Access and usability Three distinct themes were identiﬁed that informed what access and usability meant to the participants: suitable alternative, Themes for Equipment technical constraints, and visual issues. One theme related to the need for additional equipment to hold and position the device for clearer visibility was identiﬁed: Suitable alternative. Suitable alternative was a theme expressing portability. that access and usability of the live stream technology worked well and live streaming is a ﬂexible and signiﬁcant tool that can be Portability. For the portability theme, most respondents indi- used to conduct workplace observation competency assessments. cated using a tripod as a platform for maintaining the stability of As one trainee said, their device during the live stream observation process. “Great tool used to observe skills and it may be something to “I used a large tripod which helped with the clear view of the consider to continue using in foreseeable future because it is surgery for my assessor.” convenient and efﬁcient.” However, respondents highlighted that other types of support, Similarly, clinicians also reported how useful it was for the such as ring lights, phone cases, lab work trolleys and phone trainees to continue their clinical practice during the pandemic. holders were used to support their devices during the process. “It was ﬁne and enabled things to continue despite the current “I have used a gooseneck phone holder. The phone holder is easy to secure on edge of table and it can be manipulated to what restrictions in clinical practice.” angle you want.” “[Trainee’s] observation went [well] she knew what was expected of her and the technology worked ﬁne.” Themes for Support and Feedback Five themes informing support and feedback experiences One of the positive aspects of using the live stream technology included process, visual support, educational value, technical was that whilst some respondents did report feeling anxious or support, and potential support if needed. stressed beforehand, when they began to use it, they reported feeling less anxious than they would have been in a face-to-face Process. Process was expressed as clear communication support scenario: provided by the assessors during the live stream observation. It was expressed by one trainee as: “I was very worried at the thought of the observation process when I started the course, but I think doing it via livestream “Because I was ﬁrst student for observations, I didn’t get the actually made it less stressful and nerve wracking.” document early enough before my observation, but I have been explained by my tutor verbally prior to my observation on the very same day.” Technical constraints. Technical constraints were described as the main challenge encountered and identiﬁed as a barrier Another clinician described this as an experience that left the to access and usability of the life stream technology. trainee fully supported during the observation and seems to Network issues, data usage and battery draining were express this theme well. regarded as problematic, especially as they were outside of the assessor’s control. However, some respondents highlighted “Overall, our trainee has been given the highest standard of their conﬁdence in and positive experiences in using the live support. Where our trainee struggled, there was always an stream: assessor there for guidance and support.” “Had no problems at any stage, felt conﬁdent all elements were able to be observed and assessed properly.” Another respondent Visual support. The theme of visual support was expressed by emphasised that: “I was nervous in case there was a drop in respondents as experiences of knowing when the assessor does signal which would interrupt observation but found that not have clear visibility. In this case, they can quickly inform the all assessments took place well without too many difﬁculties.” trainees about the visual support to enhance visibility. This theme BDJ Open (2023) 9:4 C. Taylor et al. encompassed several attributes including trainees’ and clinicians’ voiced concerns about technical problems such as network issues ability to know what to do when visibility is reduced. One trainee causing frequent disruption in the observation ﬂow; system issues shared her experience of increasing visibility during the observa- such as high-data usage and fast draining of battery; and tion process: infrastructural problems such as less visibility where assessors struggle to see clearly, and less value in using the live stream “During one of the observations my assessor didn’t have a clear technology appeared in the response for overall satisfaction. view and made me aware of this which was sorted accordingly Respondents thought that some of the challenges were com- with the help of the clinician and assessor.” pounded by the type of workplace building such as old buildings with thicker walls that restricted internet ﬂow. A trainee noted, “… When connection wasn’t very good it became Educational value. Educational value was a third theme in quite easy to get stressed to one ensure time keeping met while still support and feedback experiences and was described as trying to get good connection and also in case it got disconnected experiences of interpersonal interactions during the observation throughout.” process. These were characterised by perceptions of beneﬁcial Arranging speciﬁc patients for the observation due to busy support, good, helpful, and useful information, and feedback clinical lists was seen as a problem by clinicians, and the live towards the trainees and clinicians by the assessor. These observation was thought to be less useful. experiences were felt to enhance the individual’s competency. A clinician shared, “Due to Covid, live stream was the only way of This theme included attributes such as feeling comfortable and getting some form of observation, though there’s not much the conﬁdent, ﬂexibility of the assessor, being able to freely contact assessor can adequately see to assess from a single, ﬁxed vantage the assessor, and feeling good about the observation process point. It was difﬁcult to try to arrange speciﬁc patients in a row for because they were supported. One trainee highlighted how an the observations due to having a busy clinical list…I’m not assessors’ support led to a successful observation: convinced the live observation was useful in our nurse’s training.” “At the start of the observation the link wouldn’t allow me to join, Positive views for overall expectation. Some positive views of my assessor was able to support me in giving me more help to using the live stream technology included ease of using the live access the livestream.” stream technology; how less stressful it was in comparison to face- to-face assessment; how well trainees coped; the ability to Educational value was seen as signiﬁcant to support and complete the observation on time; how conﬁdent trainees felt in feedback in the live stream observation process, as expressed in being observed virtually; ease of contacting the assessor; and clear the words of one of the trainees: “I enjoyed the live stream visibility. The predominant positive view reported by both trainees observation because the feedback I received made me feel conﬁdent and clinicians was how the live stream technology worked well to in the fact that I was doing my job well and I will make a good ensure the assessment of trainees was performed during the dental nurse.” pandemic. One trainee shared: “It was good overall as was quite ﬂexible.” Technical support. Technical support as a theme represented Another trainee highlighted, “Live Streaming observation was difﬁculties experienced due to technological interferences of positive in the way that due to the current circumstances I was still internet connectivity and troubleshooting technology support able to be observed in my work, there was a couple of times due to working in a hospital setting it required slightly more planning but provided by the assessor. Respondents shared many experiences of how they were supported during technology problems and overall, I was happy I was able to continue with my course and do how personal examples of the assessor made a difference. One my observations due to the circumstances.” trainee shared, It was recommended that recording and compiling individual treatment would be much easier. Nevertheless, despite these “I had problem with connecting my integrated camera and my concerns other respondents also expressed that the use of the live tutor was telling me about her personal experience and how she stream technology was an appropriate alternative. This is shown in sorted out that problem. I was very stressed about it and helped some clinicians’ responses: me out a lot to calm down.” “Overall fantastic”. “I think the process of live stream assessment generally worked well in this scenario.” Potential support if needed. The last theme, potential support if needed, was expressed as being able to receive technological support if required but it was not needed. One clinician stated, ASSESSOR PERCEPTION FOR EFFECTIVELY ASSESSING “I personally did not require technology support; however, our PERFORMANCE USING THE LIVE STREAM TECHNOLOGY trainee was supported with any technical issues, and these were To effectively assess performance using live stream technology, resolved.” the factors which shaped the assessors’ perceptions have been investigated. In this case assessors were asked to comment on whether the live stream observations enabled them to effectively Themes for Expectation [Performance/Overall Satisfaction] assess the performance of trainee dental nurses. Assessors’ Two themes emerged for overall satisfaction with using the live perceptions of the effectiveness of using the live stream stream technology: negative and positive views for expectation. technology varied, and mixed responses were received. The Negative views mean respondents experienced some challenges ﬁndings have been grouped into two categories: factors aiding when using the live stream technology and were not satisﬁed. effectiveness and factors inhibiting effectiveness. Factors aiding Positive views, on the other hand mean that respondents effectiveness were efﬁciency, self-efﬁcacy, clear visibility, cost- experienced some valued beneﬁt when using the live stream effectiveness, and reduced travel time. Factors inhibiting effective- technology and were satisﬁed. ness were technical glitches (Wi-Fi connection, high data usage and sound quality), limited visibility, ﬁrst time use of digital Negative views for overall expectation. Some negative views of observation, and preferred method of observation (face-to-face/ using the live stream technology were identiﬁed. Respondents blended approach). BDJ Open (2023) 9:4 C. Taylor et al. Most assessors highlighted ‘some sort of [effective] strength’ during face-to-face observations); Achievements (Ability to associated with using the live stream technology. One assessor support trainees with better visibility during the observation said, “Most candidates coped well, and you were able to see enough process e.g., using phone holders that can hang to the wall; to note.” Another assessor expressed that he or she: “Was able to ability to complete qualiﬁcations during a pandemic and carry out effective observation of candidates performance during a upskilling of assessors by embracing this new environment to range of clinical dental procedures.” conduct the workplace assessment). The method used for the observation was felt to be all important. Another assessor emphasised on the tone of cost saving that: “There are a lot of beneﬁts no travel time or costs, if DISCUSSION patient cancels no time wasted for assessor, candidate can set up an This study contributes to knowledge in dental education as it observation last minute with assessor.” Being a suitable alternative reports on an innovative strategy that has the potential to change was also viewed as equally signiﬁcant: “Live stream observations the way in which workplace assessment is conducted. Consistent are certainly an obvious alternative to in-practice observations.” In with previous research into digital technologies for assessment addition, “Live stream observations have worked well. Initially there e.g. [1, 13–15, 22], we found that the beneﬁts of using live stream was some reluctance from practitioners and there has been slight technology for conducting the workplace observation assessment issue with sound at times but apart from that all good.” signiﬁcantly outweighed the challenges encountered. This use of However, several assessors questioned the effectiveness of digital technology strategy seems to enhance sustainable teach- using the live stream technology, discussing factors that inhibited ing, learning and assessment particularly during the COVID-19 its use. The difﬁculty of measuring effectiveness was raised with pandemic. The implementation of this innovative method has concerns of technical glitches, most assessors voiced concerns been successful and enabled assessment and progression of pre- regarding WIFI connection, data usage, sound quality, and limited registration dental nurses throughout the COVID-19 pandemic. visibility. One assessor highlighted, “For some yes, but others no, The successful implementation required assessors who created a very much depended on view and WIFI connection.” Nevertheless, consistent approach and supported the trainees to be familiar while some felt that there were some inhibiting factors of with the process. In addition, removing the requirement for an effectively assessing the trainees, others reserved criticism on the assessor to travel to visit a trainee’s workplace to conduct the basis of unprecedented circumstances that could be resolved. observation assessment and overall, signiﬁcantly reduced time taken to undertake this assessment process and cost of travel. “It can be difﬁcult to see everything and is connection dependant, Overall providing a more ﬂexible, sustainable approach to but I set up the meeting before the appointment time so that I arranging this method of assessment. To the best of our can observe the candidate setting up for the procedure and I ask knowledge, this is the ﬁrst study to evaluate the experiences of them to pan round the surgery. This also gives me the trainees’, clinicians, and assessors’ together on the use of live opportunity to ask any questions therefore fewer questions at stream observation. the end of the observation if the student does not have much The reliability of this study lay at an average of alpha value time.” greater than 0.6 across the instruments which is acceptable and there was a good response rate to completing the questionnaires. Another assessor expressed, “…What I was not able to capture, I The study represents a major change in workplace observation questioned my student after the procedure or arranged for a TEAMS assessment. The live stream observation was highly beneﬁcial for meeting when suitable for my candidate. Only once did I re-observe assessors as it decreased the need for in-practice assessment and the procedure.” efﬁciently compensated for the disruption of the trainees’ Among factors related to the effectiveness of using the live assessment. Overall, most trainees viewed the live stream stream technology, a preference for in person observation was a observation process positively, including preparing for the view held by some assessors, “I feel nothing can beat a face-to-face observation process, access and usability, feedback and support, observation, meeting with the student, the staff and looking at the and satisfaction. This is consistent with the ﬁndings from a study environment and building relations.” assessing undergraduate nursing students’ perspectives towards Another assessor emphasised, “It is very much depending on the video assessment of clinical skills that reported that students student, ability to get a good view of the working area and how accepted the video assessment process positively [14]. The much you can actually hear, especially regarding communication expectation component based on performance is an important with both the patient and the dentist. Still feel a blended approach is strength of using the live stream technology. a must to ﬁll in the blanks/make conﬁdent assessment decisions.” All the trainees indicated that the live stream observation enabled them to demonstrate their performance as dental nurses. The trainees were expected to demonstrate their capability for FOCUS GROUP FINDINGS FROM ASSESSORS safe and effective practice and maintain sufﬁcient ﬂexibility for We conducted three focus groups (n = 4, 4 and 5 participants, providing patient care while using live streaming. Based on our respectively) exploring assessors’ thoughts and experiences of ﬁndings, the live stream observation offered a safe working space, using the live stream observation with the following aims: ﬁrst, provided opportunity for the trainees to progress with their to identify potential information not already identiﬁed in the training, receive support from their assessors and feel conﬁdent survey; and second, to derive a meaningful assessment of user about their job. In agreement, most clinicians (82.9%) indicated experience in live stream observation as a group. The analysis of that the live stream observation enabled their trainees to the focus groups identiﬁed three core areas of reﬂections demonstrate their performance as a dental nurse. This contrasts relevant to using the live stream technology for workplace with the report of [15] who highlighted some concerns such as assessment: Beneﬁts (Convenience such as ability to organise nerves and performance being affected by using video assess- observations in limited time, ease of giving feedback, ﬂexibility ment but indicated its usefulness for assessing competency with an opportunity to build relationship with missed dental regards to feedback. However, our ﬁndings align with Kam and practices; Time effectiveness and cost effectiveness such as others (2019) who reported increased performance competence increased time for trainees to ask questions, reduced travel time of core basic nursing skill among participants that received video and possibility of short notice observation; reduced pressure); feedback [17]. In this study, the trainees also shared how they had Challenges (Technical difﬁculties such as poor internet connec- learnt from the personal experience of the assessors to remain tions, sound quality and less visibility; losing skills observed conﬁdent during the assessment process. BDJ Open (2023) 9:4 C. Taylor et al. The live stream observation therefore allowed the trainees to disciplines in the implementation of using technology to conduct present to the assessors how their practical experiences at the workplace assessment. The ﬁndings from this study and further practice had culminated into new insights, technological experi- analysis will inform where continual improvements can be made ences, and new learning. In addition, all the trainees reported their to effectively support students and workplace teams in the overall satisfaction experience with using the live stream was organisation, and facilitation of remote workplace observation good or excellent. Consequently, clinicians also expressed high competency-based assessment via live video stream. levels of overall satisfaction with using the live stream technology. Furthermore, (86.7%) of assessors reported that they had a good or very good overall satisfaction experience with the live stream CONCLUSIONS, RECOMMENDATIONS, AND FUTURE WORK observation. This is a signiﬁcant transformative outcome for Our main contribution in this paper is to investigate the views and technology-based workplace assessment in dentistry. Guze (2015) experiences of using live streaming for workplace observation highlighted the transformative learning experiences associated assessment to assess the overall satisfaction and effectiveness. with using technologies that engages the learner and Overall, it was found that satisfaction of using live streaming can be allows learning that does not endanger the patient [28]. It is linked with the resolving technology issues and support received, but also expected that assessors should be able to support effectiveness is mixed. Our results, as well as the limitations of learning into a more collaborative, personalised, and empowering supporting data, point naturally to several directions for future work. experience [28]. However, sustaining this method and the future use of remote live Across the three groups, median overall satisfaction rates were observation will be dependent on the Skills for Health decision to higher for trainees and clinicians than assessors (4.0, 4.0 and 3.0, maintain this as an assessment method with the assessment strategy respectively), which represents good to very good experience for the SVQ in Dental Nursing qualiﬁcation. Nevertheless, the use of among respondents. In addition, the median effectiveness score remote live observation assessed a range of generic and practical for assessors was 4.0, which shows that the assessors agreed with trainee competencies that require a higher level of conﬁdence in effectively assessing the performance of the trainees using the live managing the technology used and carrying on with the assessment. stream technology. These results suggest that trainees, clinicians, It allowed for feedback and validation of the trainee learning and assessors had a positive experience of using live stream experiences and was viewed as an efﬁcient and effective way of observation for the workplace observation assessment, and its use assessing and providing feedback particularly in response to global had an effective impact on trainee performance outcome and pandemics such as the changing condition posed by the ongoing achievement. Given the changing situation, as this method of COVID-19 pandemic. Thus, it is a feasible option for assessing trainees assessment is increasingly used, the assessors will likely better based in areas with transport difﬁculty (for example, practices in the understand its beneﬁt to improving trainees’ performance out- Western Isles) without having to travel. comes and reﬂective learning and their satisfaction rate will reﬂect NES has created a vision for Technology Enhanced Learning supporting learning effectiveness. that will enable education by facilitating appropriate, high-quality However, a key challenge in using live streaming is the technology Technology-Enhanced Learning for the Health and Social Care issue around internet stability which a concern for the trainees, workforce now and in the future. The ﬁndings and learning from clinicians, and assessors, as some were disconnected during the this project will be shared to support the integration and staff observation process. This may affect their abilities to be coordinated development of utilising the functionalities of various technolo- when performing the task during the observation process. Similarly, gies within blended learning programmes as a robust assessment using live streaming can constitute physical barriers for trainees who tool. Therefore, as the use of remote live observation continues to may be perplexed by their requirements, structure, and navigation. be developed, consideration should be given to incorporate Nevertheless, to combat this technology issues, before the observa- features that enable acceptance and re-imbursement for high- tion, a guidance document on this method of assessment and speed internet/phone expenses. Further local research and support for vital preparation was sent to the trainees and during the comparisons will help to identify practical and efﬁcient ways of assessment process, the assessors supported the trainees with a using live streaming to conduct workplace-based assessment and variety of troubleshooting techniques. The trainees, however, were give feedback to encourage learning among dental trainees. positive about using live stream observation as a tool for their Nevertheless, the use of the live stream technology generally workplace assessment stating that the guidance document provided worked well for the purpose of the workplace-based competency helped them prepare to use method for the assessment, allowed assessment. What was learned from the experience with the them to arrange a suitable time with their assessor and worked in a COVID-19 pandemic is the need for ﬂexibility concerning less stressful and nerve-wracking environment, even when visibility conducting the workplace-based assessment. was reduced. This is because the assessors made them aware they didn’t have a clear view and advised on best position to enhance REFERENCES visibility, which made the observation process run smoothly. 1. Rao S, Chowdhury N, Naithani M, Singh A, Singh N, Gupta AK. et al. Alternative strategy adopted in practical assessment for pathology postgraduate certiﬁcation examination amidst COVID-19 Nationwide lockdown: an Indian experience with a STUDY LIMITATIONS AND STRENGTH novel hybrid method of assessment. Adv. Med. Educ. Pract. 2020;12:1 https:// The limitation of the study is centred around the limited published www.ncbi.nlm.nih.gov/pmc/articles/PMC7802903/. data to support the use of live stream observation as a method of 2. El-Kishawi M, Khalaf K, Al-Najjar D, Seraj Z, Al Kawas S. Rethinking assessment workplace-based assessment. As this study involves a small concepts in dental education. Int. J. Dent.2020;2020:8672303 https:// number of dental nurse trainees, the ﬁndings may not be a www.ncbi.nlm.nih.gov/pmc/articles/PMC7584943/pdf/IJD2020-8672303.pdf. reﬂective of the personal experiences and viewpoints of all dental 3. Williams JC, Baillie S, Rhind SM, Warman S, Sandy J, Ireland A. A Guide to nurse trainees, clinicians, and assessors of other training providers. Assessment in Dental Education. 2015. https://research-information.bris.ac.uk/en/ publications/a-guide-to-assessment-in-dental-education. However, a key strength of the study is that it investigated views 4. Grieveson B, Kirton JA, Palmer NOA, Balmer MC. Evaluation of workplace based and experiences of using live stream observation from different assessment tools in dental foundation training. Br Dent J.2011;211:E8–E8. https:// perspectives and took a mixed methods approach to validate the www.nature.com/articles/sj.bdj.2011.681.pdf. ﬁndings. Moreover, this research indicates that this method 5. Kalsi HK, Kalsi JS, Fisher NL. An explanation of workplace-based assessments in was effective, and support from NES’s DCP workstream team postgraduate dental training and a review of the current literature. Br Dent signiﬁcantly contributed to the successful implementation. It is J.2013;215:519–24. https://www.nature.com/articles/sj.bdj.2013.1098.pdf. hoped that this paper can support assessors in dentistry and other BDJ Open (2023) 9:4 C. Taylor et al. 6. Manekar VS, Radke SA. Workplace based assessment (WPBA) in dental education- 28. Guze PA. Using technology to meet the challenges of medical education. Trans. A review. J Educ Technol Health Sci.2018;5:80–85. https://www.ipinnovative.com/ Am. Clin. climatological Assoc.2015;126:260. https://www.ncbi.nlm.nih.gov/pmc/ journal-article-ﬁle/7503. articles/PMC4530721/pdf/tacca1260000260.pdf. 7. Uma E. Workplace-based assessment: A valuable tool in undergraduate dental education. Int. J. Appl. Basic Med. Res.2020;10:223. https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC7758789/. ACKNOWLEDGEMENTS 8. General Dental Council. Employment and Registration. https://www.gdc-uk.org/ The authors would like to express their gratitude to all respondents who participated information-standards-guidance/supporting-the-dental-team/employment-and- in the study. We are also thankful and appreciate the help and support provided by registration#employment-and-roles-and-responsibilities-of-student-dental- the following staff: Pam Nicoll, Programme Director RRHEAL, NHS Education for nurses-and-technicians. Scotland. Alan Gillies, Knowledge Manager, NHS Education for Scotland. Dental Care 9. Skills for Health. Supplementary guidance for SVQ Dental Nursing approved centres to Professionals Workstream, NHS Education for Scotland. Dental Clinical Effectiveness support learners impacted by the COVID-19 pandemic. 2020. https://www.sqa.org.uk/ Workstream, NHS Education for Scotland sqa/ﬁles_ccc/svq-dental-nursing-supplementary-guidance-covid19.pdf. 10. Scottish Qualiﬁcations Authority. Scottish Vocational Qualiﬁcation in Dental Nursing. Available at SVQ Dental Nursing—SQA. https://www.sqa.org.uk/sqa/67607.html. AUTHOR CONTRIBUTIONS 11. Scottish Qualiﬁcations Authority. Guidance on assessment of Scottish Voca- CT conceived the study, acquisition of the data and initial draft preparation. AI tional Qualiﬁcations (SVQs). 2013. https://www.sqa.org.uk/ﬁles_ccc/ developed the methods and strategies of the data analysis, performed the data DB6083_Guidance_on_Assessment_of_SVQs.pdf. analysis, data interpretation and wrote the original draft manuscript. LY checked the 12. Brimble M. Skills assessment using video analysis in a simulated environment: an data analysis, critically revised the manuscript, and provided ﬁnal approval of the evaluation. Nursing Children and Young People. 2008;20. https://doi.org/10.7748/ manuscript. All authors reviewed and edited the ﬁnal draft manuscript. All authors paed2008.09.20.7.26.c6705. have read and agreed to the published version of the manuscript. 13. Forbes H, Oprescu FI, Downer T, Phillips NM, McTier L, Lord B. et al. Use of videos to support teaching and learning of clinical skills in nursing education. A review. Nurse Education Today 42, 53–56. Harden, R., 1988. What is an OSCE?. Med. Teach.2016;10:19–22. https://pubmed.ncbi.nlm.nih.gov/27237353/. COMPETING INTERESTS 14. Lewis P, Hunt L, Ramjan LM, Daly M, O’Reilly R, Salamonson Y. Factors con- The authors declare that they have no competing interests with respect to the tributing to undergraduate nursing students’ satisfaction with a video assess- authorship and/or publication of this article. ment of clinical skills. Nurse Educ. today. 2020;84:104244. https:// pubmed.ncbi.nlm.nih.gov/31715471/. 15. Ho RC, Song BL. Immersive live streaming experience in satisfying the learners’ ETHICS APPROVAL need for self-directed learning. Interact. Technol. Smart Educ. 2021; 10.1108/ITSE- The study was assessed using the NHS Health Research Authority deﬁning research 12-2020-0242. online decision-making tool. This tool conﬁrmed that the study would not be considered 16. Snekalatha S, Marzuk SM, Meshram SA, Maheswari KU, Sugapriya G, Sivasharan K. research by the NHS and that NHS Research Ethics Committee review was not required. Medical students’ perception of the reliability, usefulness and feasibility of The study was governed in accordance with NES’s guidance for the conduct of unproctored online formative assessment tests. Adv Physiol Educ.2021;45:84–88. evaluation studies, and institutional ethical review was also not required. https://doi.org/10.1152/advan.00178.2020. 17. Kam BS, Yune SJ, Lee SY, Im SJ, Baek SY. Impact of video feedback system on medical students’ perception of their clinical performance assessment. BMC Med INFORMED CONSENT Educ.2019;19:1–7. https://doi.org/10.1186/s12909-019-1688-6. Informed consent was obtained from all participants involved in the process: trainees, 18. Weeks DL, Molsberry DM. Feasibility and reliability of remote assessment of PALS clinicians, and assessors. The data used for the study was pseudonymised before the psychomotor skills via interactive videoconferencing. Resuscitation. 2009;80:354–8. analysis was conducted and managed according to NES privacy policy. https://www.sciencedirect.com/science/article/abs/pii/S0300957208007831. 19. Choy I, Fecso A, Kwong J, Jackson T, Okrainec A. Remote evaluation of laparo- scopic performance using the global operative assessment of laparoscopic skills. ADDITIONAL INFORMATION Surg Endosc.2013;27:378–83. https://pubmed.ncbi.nlm.nih.gov/22890477/. 20. McQueen S, McKinnon V, VanderBeek L, McCarthy C, Sonnadara R. Video-based Correspondence and requests for materials should be addressed to Caroline Taylor. assessment in surgical education: a scoping review. J. Surg Educ. 2019;76:1645–54. https://pubmed.ncbi.nlm.nih.gov/31175065/. Reprints and permission information is available at http://www.nature.com/ 21. Wang Y, Dore K, Wong N, Hoogenes J, Matsumoto E. MP34-12 Utility and feasi- reprints bility of video-based assessment of residents performing ureteroscopy and laser lithotripsy. J. Urol. 2020;203:e507–e508. https://www.auajournals.org/doi/abs/ Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims 10.1097/01.JU.0000556009.17548.ba. in published maps and institutional afﬁliations. 22. Angelo RL, Pierre PS, Tauro J, Gallagher AG. Live Observational Objective Assessment of Operative Performance in a Cadaveric Model is Equivalent to Delayed Video-Based Assessment. Arthrosc.: J. Arthroscopic Relat. Surg.2021;37:3241–7. https://pubmed.ncbi.nlm.nih.gov/33964394/. Open Access This article is licensed under a Creative Commons 23. Pugh CM, Hashimoto DA, Korndorffer JR,Jr. The what? How? And who? Of video Attribution 4.0 International License, which permits use, sharing, based assessment. Am. J. Surg.2021;221:13–18. https://pubmed.ncbi.nlm.nih.gov/ adaptation, distribution and reproduction in any medium or format, as long as you give 32665080/. appropriate credit to the original author(s) and the source, provide a link to the Creative 24. IBM Corp. IBM statistical package for social sciences (SPSS) for windows, (version Commons license, and indicate if changes were made. The images or other third party 27.0). Armonk, NY, USA: IBM Corp; 2020. material in this article are included in the article’s Creative Commons license, unless 25. Dall’Oglio AM, Rossiello B, Coletti MF, Caselli MC, Ravà L, Di Ciommo V. et al. indicated otherwise in a credit line to the material. If material is not included in the Developmental evaluation at age 4: Validity of an Italian parental ques- article’s Creative Commons license and your intended use is not permitted by statutory tionnaire. J. Paediatr Child Health. 2010;46:419–26. https://doi.org/10.1111/ regulation or exceeds the permitted use, you will need to obtain permission directly j.1440-1754.2010.01748.x. from the copyright holder. To view a copy of this license, visit http:// 26. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative creativecommons.org/licenses/by/4.0/. Res. Psychol.2006;3:77–101. https://www.tandfonline.com/doi/abs/10.1191/ 1478088706qp063oa. © The Author(s) 2023 27. Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: Striving to meet the trustworthiness criteria. Int. J. qualitative methods. 2017;16:1609406917733847 https://journals.sagepub.com/doi/full/10.1177/1609406917733847. BDJ Open (2023) 9:4

Journal

BDJ Open – Springer Journals

Published: Feb 7, 2023

References

Alternative strategy adopted in practical assessment for pathology postgraduate certification examination amidst COVID-19 Nationwide lockdown: an Indian experience with a novel hybrid method of assessment

Rao, S; Chowdhury, N; Naithani, M; Singh, A; Singh, N; Gupta, AK
Rethinking assessment concepts in dental education

El-Kishawi, M; Khalaf, K; Al-Najjar, D; Seraj, Z; Al Kawas, S
Evaluation of workplace based assessment tools in dental foundation training

Grieveson, B; Kirton, JA; Palmer, NOA; Balmer, MC
An explanation of workplace-based assessments in postgraduate dental training and a review of the current literature

Kalsi, HK; Kalsi, JS; Fisher, NL
Workplace based assessment (WPBA) in dental education-A review

Manekar, VS; Radke, SA
Workplace-based assessment: A valuable tool in undergraduate dental education

Uma, E
Use of videos to support teaching and learning of clinical skills in nursing education. A review. Nurse Education Today 42, 53–56. Harden, R., 1988. What is an OSCE?

Forbes, H; Oprescu, FI; Downer, T; Phillips, NM; McTier, L; Lord, B
Factors contributing to undergraduate nursing students’ satisfaction with a video assessment of clinical skills

Lewis, P; Hunt, L; Ramjan, LM; Daly, M; O’Reilly, R; Salamonson, Y
Medical students’ perception of the reliability, usefulness and feasibility of unproctored online formative assessment tests

Snekalatha, S; Marzuk, SM; Meshram, SA; Maheswari, KU; Sugapriya, G; Sivasharan, K
Impact of video feedback system on medical students’ perception of their clinical performance assessment

Kam, BS; Yune, SJ; Lee, SY; Im, SJ; Baek, SY
Feasibility and reliability of remote assessment of PALS psychomotor skills via interactive videoconferencing

Weeks, DL; Molsberry, DM
Remote evaluation of laparoscopic performance using the global operative assessment of laparoscopic skills

Choy, I; Fecso, A; Kwong, J; Jackson, T; Okrainec, A
Video-based assessment in surgical education: a scoping review

McQueen, S; McKinnon, V; VanderBeek, L; McCarthy, C; Sonnadara, R
MP34-12 Utility and feasibility of video-based assessment of residents performing ureteroscopy and laser lithotripsy

Wang, Y; Dore, K; Wong, N; Hoogenes, J; Matsumoto, E
Live Observational Objective Assessment of Operative Performance in a Cadaveric Model is Equivalent to Delayed Video-Based Assessment

Angelo, RL; Pierre, PS; Tauro, J; Gallagher, AG
The what? How? And who? Of video based assessment

Pugh, CM; Hashimoto, DA; Korndorffer, JR
Developmental evaluation at age 4: Validity of an Italian parental questionnaire

Dall’Oglio, AM; Rossiello, B; Coletti, MF; Caselli, MC; Ravà, L; Di Ciommo, V
Using thematic analysis in psychology

Braun, V; Clarke, V
Thematic analysis: Striving to meet the trustworthiness criteria

Nowell, LS; Norris, JM; White, DE; Moules, NJ
Using technology to meet the challenges of medical education

Guze, PA

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience

Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience

Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience

Abstract

Journal

Recommended Articles

References

Our policy towards the use of cookies