Testing the Construct Validity of a Virtual Reality Hip Arthroscopy Simulator
Vikas Khanduja, M.A., M.Sc., F.R.C.S.(Orth), John E. Lawrence, M.A., M.B., BChir, M.R.C.S., and Emmanuel Audenaert, M.D., Ph.D.
With an ever-increasing number of diagnostic and therapeutic indications, hip arthroscopy is one of the most rapidly expanding areas of modern surgery.1-4 The ball-and-socket nature of the joint, together with the thickness of the capsule and soft-tissue envelope, make it a technically demanding field with a steep learning curve.2 An estimated 30 training cases are required before a reduction in complications is observed.5 Such complications include labral or chondral injury from instrumentation, bleeding or nerve injury from portal insertion, and traction-related neurapraxia. 6 It is therefore vital that orthopaedic surgeons gain exposure to arthroscopic hip surgery early in their training and quickly develop the basic skill set necessary to progress towards competency.
Although the traditional model of the apprenticeship in surgical training still applies today, the evolution of more technically demanding surgical techniques combined with a reduction in the trainee’s caseload has led to steep learning curves in the modern era.7-11 Several studies have highlighted the length of time taken to become a safe and effective arthroscopic surgeon, demonstrating both increased operative time and a greater incidence of complication when an inexperienced clinician is performing arthroscopic hip surgery.12-14 Multiple studies have also suggested a role for virtual reality (VR) simulation in overcoming this steep learning curve by equipping the orthopaedic trainee with the basic skill set required to safely perform arthroscopic procedures.15-17 In the context of hip arthroscopy, improving skills with simulation before performing a procedure in the operating room has the potential to reduce those complications associated more closely with surgical technique, such as labral and chondral injury.
Validation of a VR hip arthroscopy simulator as an evaluation tool is required before it can be used to this end. The purpose of this study was to test the construct validity of the hip diagnostics module of a VR hip arthroscopy simulator by comparing novice and expert performance over 2 tasks of increasing difficulty. The hypothesis was that expert arthroscopic hip surgeons would achieve superior performance on the simulator compared with those with little experience, supporting the simulator’s construct validity.
Virtual reality hip arthroscopy simulator demonstrates sufficient face validity
Jonathan D. Bartlett1 · John E. Lawrence2 · Vikas Khanduja2
Received: 19 December 2017 / Accepted: 25 June 2018
Over the last decade, there has been increasing investigation into the use of virtual reality (VR) simulation in surgical education . The technically demanding nature of hip arthroscopy, combined with reductions in trainee operating time, has led to steep learning curves in modern orthopaedic surgery [2, 6, 7, 14, 15, 18, 20]. Additionally, multiple studies have highlighted worse outcomes and increased operating time in arthroscopic operations performed by inexperienced surgeons, demonstrating higher incidences of chondral damage and perineal injuries in hip arthroscopies performed in the first 3 years of an arthroscopic surgeon’s training [9, 18, 21]. These increased complication rates and longer operating times mandate a strategy to overcome this initial learning curve.
Training young adult hip surgeons for the future: the Cambridge vision
Karadi H. Sunil Kumar MRCSEd, MCh Orth, Dip (SEM), FEBOT
Specialty Registrar in Trauma & Orthopaedics, Addenbrooke’s
Hospital, Cambridge University Hospitals NHS Foundation Trust, UK
John E. Lawrence MA (Cantab), MB BChir, MRCS
Core Surgical Trainee in Trauma & Orthopaedics, Addenbrooke’s
Hospital, Cambridge University Hospitals NHS Foundation Trust, UK
Vikas Khanduja MA (Cantab), MRCS (G),
MSc, FRCS, FRCS (Orth)
Consultant Orthopaedic Surgeon,
Addenbrooke’s Hospital, Cambridge University
Hospitals NHS Foundation Trust, UK
Bone & Joint360, volume 5, issue 6, December 2016
Surgery is a form of art which has transformed over the last six centuries from an individual’s craftsmanship to a team-based approach to ensure patient safety.1 Surgical training has also seen a compelling change from being a primarily apprenticebased training to a competency-based model. Although the traditional model of apprenticebased
training is still valid in modern surgical training, there has been a gradual shift over the last two decades towards achieving specific competencies in an objective manner to deliver safe surgical care.2,3 There are several reasons for this change. A reduction in working hours in the UK and Europe, as dictated by the European Working
Time Directive (EWTD), has reduced the total training time between qualification and becoming a consultant (specialist) from 30 000 hours to 6000 hours.4 This five-fold reduction in hours has not only reduced the amount of surgical exposure gained by the resident during training, but has also resulted in significant variability in the orthopaedic caseload for the trainee.5-8 In parallel with this, the introduction of consultant-delivered care in an effort to improve patient outcomes may have an indirect effect on surgical training by reducing the number of cases a resident performs as primary surgeon.9 A final driving force for the shift in curriculum is the continued evolution of technically demanding surgery with steep learning curves, such as arthroscopy, which is now increasingly performed in orthopaedic
practice due to several technical advances.
Arthroscopic Training Courses Improve Trainee Arthroscopy Skills: A Simulation-Based Prospective Trial
Evans Army Community Hospital, Fort Carson, Colorado, U.S.A.. Electronic address: Dr.firstname.lastname@example.org.
DMC Sports Medicine, Detroit Medical Center, Detroit, Michigan, U.S.A.
The John A. Feagin Jr Sports Medicine Fellowship, Keller Army Hospital, US Military Academy, West Point, New York, U.S.A.
Arthroscopy: The Journal of Arthroscopic and Related Surgery, March 2016
PURPOSE: To evaluate the correlation between timed task performance on an arthroscopy shoulder simulator and participation in a standardized expert shoulder arthroscopy educational course.
METHODS: Orthopaedic trainees were voluntarily recruited from over 25 residency programs throughout the United States and Canada. Each trainee was tested on arrival at the Arthroscopy Association of North America orthopaedic learning center on a virtual reality arthroscopy shoulder simulator, and his or her performance was objectively scored. Each trainee’s postgraduate year level was recorded, as was his or her experience in residency with shoulder arthroscopy as measured by Accreditation Council for Graduate Medical Education case-log totals. After the focused 4-day training curriculum consisting of didactics and cadaveric experience, each trainee was re-evaluated on the same simulator. Statistical analysis was performed to determine if participation in the course was associated with changes in simulation performance from before to after assessment.
RESULTS: Forty-eight trainees completed the testing. On completion of the course, trainees showed significant improvements in all objective measures recorded by the simulator. Total probe distance needed to complete the task decreased by 42% (from 420.4 mm to 245.3 mm, P < .001), arthroscope tip distance traveled decreased by 59% (from 194.1 mm to 80.2 mm, P < .001), and time to completion decreased by 38% (from 66.8 seconds to 41.6 seconds, P < .001). Highly significant improvements in all 3 measures suggest improved instrument handling, anatomic recognition, and arthroscopy-related visual-spatial ability.
CONCLUSIONS: This study shows objective improvement in orthopaedic trainee basic arthroscopy skill and proficiency after a standardized 4-day arthroscopy training curriculum. The results validate the Arthroscopy Association of North America resident training course and its curriculum with objective evidence of benefit.
LEVEL OF EVIDENCE: Level III, prospective study of nonconsecutive participants.
Arthroscopy Skills Development with a Surgical Simulator
A Comparative Study in Orthopaedic Surgery Residents
Brian J. Rebolledo,*y MD, Jennifer Hammann-Scala,z CST, Alejandro Leali,y MD,
and Anil S. Ranawat,y MD
Investigation performed at the Hospital for Special Surgery, New York, New York, USA
BACKGROUND: Surgical simulation has become increasingly relevant to orthopaedic surgery education and could translate to improved operating room proficiency in orthopaedic surgery trainees.
Purpose: To compare the arthroscopic performance of junior orthopaedic surgery residents who received training with a knee and shoulder arthroscopy surgical simulator with those who received didactic training.
Study Design: Controlled laboratory study.
Methods: Fourteen junior orthopaedic surgery residents at a single institution were randomized to receive knee and shoulder arthroscopy training with a surgical simulator (n = 8) or didactic lectures with arthroscopy models (n = 6). After their respective training, performance in diagnostic knee and shoulder arthroscopy was assessed using a cadaveric model. Time to completion and assessment of arthroscopic handling using a subjective injury grading index (scale, 1-10) was then used to evaluate performance
in final cadaveric testing.
Results: Orthopaedic surgery residents who trained with a surgical simulator outperformed the didactic-trained residents in shoulder arthroscopy by time to completion (–35%; P = .02) and injury grading index (–35%; P = .01). In addition, a trend toward improved performance of knee arthroscopy by the simulator-trained group was found by time to completion (–36%; P = .09) and injury grading index (P = .08).
Conclusion: In this study, junior orthopaedic surgery residents who trained with a surgical simulator demonstrated improved arthroscopic performance in both knee and shoulder arthroscopy. However, future validation of surgical simulator training for orthopaedic surgery residents remains warranted.
Clinical Relevance: Surgical skill development with an arthroscopy surgical simulator could translate to improved arthroscopy performance in the operating room.
Keywords: virtual reality; surgical simulator; surgical education; arthroscopy simulator
Testing Basic Competency in Knee Arthroscopy Using a Virtual Reality Simulator
Exploring Validity and Reliability
Mads Emil Jacobsen, MS; Morten Jon Andersen, MD; Claus Ol Hansen, MD; Lars Konge, MD, PhD
J Bone Joint Surg Am, 2015 May 06; 97 (9): 775 -781 . http://dx.doi.org/10.2106/JBJS.N.00747
Background: Diagnostic knee arthroscopy is a common procedure that orthopaedic residents are expected to learn early in their training. Arthroscopy requires a different skill set from traditional open surgery, and many orthopaedic residents feel less prepared for arthroscopic procedures. Virtual reality simulation training and testing provide an opportunity to ensure basic competency before proceeding to supervised procedures in patients.
Methods: Twenty-six physicians (thirteen novices and thirteen experienced arthroscopic surgeons) were voluntarily recruited to perform a test consisting of five arthroscopic procedures on a knee arthroscopy simulator. Performance was evaluated by obtaining predefined metrics from the simulator for each procedure, and z-scores, describing suboptimal performance, were calculated from the metrics. The intercase reliability of the simulator metrics was explored by calculating an intraclass correlation coefficient. Finally, a pass-or-fail standard was set with use of the contrasting groups method, and the consequences of the pass-or-fail standard were explored.
Results: One procedure was excluded from the final test because of a lack of validity. The total Z-scores for the four procedures included in the final test showed an intercase reliability of 0.87 (95% confidence interval, 0.78 to 0.93). The total mean z-score (and standard deviation) was 38.6 ± 27.3 points for the novices and 0.0 ± 9.1 points for the experienced surgeons (p < 0.0005). The pass-or-fail standard was set at a total z-score of 15.5 points, resulting in two of the novices passing the test and a single experienced surgeon failing the test.
Conclusions: By combining four procedures on a virtual reality arthroscopy simulator, it was possible to create a valid, reliable, and feasible test of basic arthroscopic competency and to establish a credible pass-or-fail standard.
Clinical Relevance: The simulation-based test and pass-or-fail standard could aid in assessing and ensuring basic competency of future orthopaedic residents before proceeding to supervised procedures in patients.
Investigation performed at the Centre for Clinical Education (CEKU) at Rigshospitalet, Copenhagen, Denmark
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. None of the authors, or their institution(s), have had any financial relationship, in the thirty-six months prior to submission of this work, with any entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also, no author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
The following abstract and poster were presented at annual American Academy of Orthopedic Surgeons meeting, AAOS, March 11-15, New Orleans, Louisiana
Transfer of Surgical Skills: the Importance of Arthrosimulation Training for Orthopaedic Surgery Residents
Marie Diane Isabelle Mousseau, cd, MDCM, MSc, Michelle Laprade, MD student, MSc,
Laurence Marck, MIng, Véronique Godbout, MD, MA, FRCSC
McGill Univesity, Universite de Montreal, Canada
Shoulder arthroscopy simulator performance correlates with resident and shoulder arthroscopy experience.
Martin KD, Cameron K, Belmont PJ, Schoenfeld A, Owens BD.
William Beaumont Army Medical Center, El Paso, TX 79920, USA.
J Bone Joint Surg Am. 2012 Nov 7;94(21):e160.
BACKGROUND: The technical skills required to perform arthroscopy are multifaceted and require supervised training and repetition. Obtaining this basic arthroscopic skill set can be costly and time-consuming. Simulation may represent a viable training source for basic arthroscopic skills. Our goal was to evaluate the correlation between timed task performance on an arthroscopic shoulder simulator and both resident experience and shoulder arthroscopy experience.
METHODS: Twenty-seven residents were voluntarily recruited from an orthopaedic residency program. Each subject was tested annually for three consecutive years on an arthroscopic shoulder simulator and objectively scored on time to completion of a standardized object localization task. Each subject’s total number of shoulder arthroscopies, all arthroscopies, and cases were calculated according to postgraduate year from their Accreditation Council for Graduate Medical Education (ACGME) case log. Generalized estimating equation multivariate regression analysis was performed to determine the correlation between simulation performance and total numbers of shoulder arthroscopies, all arthroscopies, and cases.
RESULTS: Univariate analyses revealed that postgraduate year, total number of shoulder arthroscopies, total number of arthroscopies of any joint, and total number of surgical cases performed during residency training prior to testing were associated with the mean time required to complete the simulator task. The number of prior shoulder arthroscopies performed (r = 0.55) and postgraduate year in training (r = 0.60) correlated most strongly with simulator basic task performance. In the multivariate analysis, the number of prior shoulder arthroscopies and postgraduate year remained independent predictors of faster completion of the simulator task. For every additional postgraduate year, there was a sixteen-second improvement in the time required to complete the simulator task (p < 0.005). Similarly, after controlling for the influence of postgraduate year, there was a twelve-second decrease in the time to complete the simulator task for every additional fifty shoulder arthroscopies performed during residency training (p < 0.008).
CONCLUSIONS: These results showed a significant relationship between performance of basic arthroscopic tasks in a simulator model and the number of shoulder arthroscopies performed. The data confirmed our hypothesis that simulator performance is representative of both resident experience and shoulder arthroscopy experience.
CLINICAL RELEVANCE: This study suggests that greater resident clinical experience and shoulder arthroscopy experience are both reflected in improved performance of basic tasks on a shoulder simulator. These findings warrant further investigation to determine if training on a validated arthroscopic shoulder simulator would improve clinical arthroscopic skills.
Arthroscopic basic task performance in shoulder simulator model correlates with similar task performance in cadavers.
Martin KD, Belmont PJ, Schoenfeld AJ, Todd M, Cameron KL, Owens BD.
William Beaumont Army Medical Center, El Paso, TX 79920, USA.
J Bone Joint Surg Am. 2011 Nov 2;93(21):e1271-5.
BACKGROUND: Attainment of the technical skill necessary to safely perform arthroscopic procedures requires the instruction of orthopaedic surgery residents in basic arthroscopic skills. Although previous studies involving shoulder arthroscopy simulators have demonstrated a correlation between task performance and the level of prior arthroscopic experience, data demonstrating the correlation of simulator performance with arthroscopic skill in a surgical setting are scarce. Our goal was to evaluate the correlation between timed task performance in an arthroscopic shoulder simulator and timed task performance in a cadaveric shoulder arthroscopy model.
METHODS: Subjects were recruited from among residents and attending surgeons in an orthopaedic surgery residency program. Each subject was tested on an arthroscopic shoulder simulator and objectively scored on the basis of the time taken to complete a standardized object selection program. After an interval of at least two weeks, each subject was then tested on a cadaveric shoulder arthroscopy model designed to replicate the shoulder arthroscopy simulator testing protocol, and the time to completion was again recorded. Both testing protocols involved the simple task of placing a probe on a series of assigned locations in the glenohumeral joint. Spearman rank correlation analysis was performed, and regression analysis was used to determine the predictive ability of the simulator score.
RESULTS: The performance time on the simulation program was strongly correlated with the performance time on the cadaveric model (r = 0.736, p < 0.001). The time required to complete the simulator task was a significant predictor of the time required to complete the cadaveric task (t = 4.48, p < 0.001).
CONCLUSIONS: These results demonstrated a strong correlation between performance of basic arthroscopic tasks in a simulator model and performance of the same tasks in a cadaveric model.