Virtual Reality and Gamification for Assessing Technical Aptitude, Cognitive Abilities, and Personality Characteristics in Surgical Residency Selection: Validation Study

Abstract

Abstract Background Assessment of technical aptitude, cognitive abilities, and personality characteristics is important in selecting candidates for surgical training. Currently, the selection of surgical training candidates does not systematically include objective assessment of these variables. Instead, it relies heavily on traditional selection methods, such as academic achievement, letters of recommendation, and interviews, whose presumed relationships with later performance are based on limited and inconsistent evidence. Objective This study examined evidence for validity based on relationships with other variables—a key source of validity evidence—to support the use of 2 novel tools for objectively assessing surgical training candidates: a nonimmersive (screen-based) virtual reality laparoscopic technical aptitude test, and a game-based assessment of cognitive abilities and personality characteristics. Methods This study had 3 phases, all focused on establishing validity based on relationships with other variables. In Phase 1, we evaluated convergent and discriminant evidence of validity by assessing the correlation between interns’ performance in the 2 novel selection tests and in 4 established psychometric instruments for assessing dexterity, visuospatial ability, intelligence, and personality. In Phase 2, we evaluated evidence for the test-criterion relationship by assessing the correlation between residents’ performance in the 2 novel tests and their concurrent residency performance evaluations. In this phase, we also assessed evidence for the fairness of the tests between genders. In Phase 3, we focused on the technical aptitude test and evaluated evidence for its relationship with training level by administering the test to a sample of expert surgeons and comparing their performance with that of the residents and interns from the previous phases. Results Interns’ scores on the 2 novel selection tests were correlated with scores on the relevant established psychometric instruments, providing convergent and discriminant evidence (Phase 1). Residents’ scores on the 2 novel tests were significantly correlated with relevant performance criteria (Phase 2). In addition, no evidence for gender bias in the tests was found. Finally, based on data collected in all 3 phases, we found evidence for expert-novice differences in the technical aptitude test scores, such that scores were correlated with surgical experience. Conclusions The findings provide validity evidence supporting the use of the novel virtual reality–based technical aptitude test and a game-based assessment of cognitive abilities and personality characteristics in selecting candidates for surgical training. However, because the test-criterion evidence was obtained using a concurrent design, further prospective longitudinal studies are required to determine whether these assessments predict subsequent residency performance.