Quality Improvement of an AI System for Determining Pass-Fail in the Fundamentals of Laparoscopic Surgery: Accuracy on a Cohort of New Users

Abstract

Surgical residents must pass the fundamentals of laparoscopy surgery test to proceed with their training. While simulation gives them an environment to practice outside the operating room, it lacks supervision. To fill this gap, we recently proposed an AI system that evaluates pass-fail in the fundamentals of laparoscopic surgery. In this quality improvement study, we sought to evaluate the model’s accuracy when detecting a failure. To do this, we performed software testing on a cohort of high school students. The students were asked to conduct the essential peg transfer FLS task under the supervision of our AI system, which evaluates them in real-time as they perform the task. Out of 18 students, the system correctly predicted the student error in 13 cases. The model must catch up on student errors due to underlying model mispredictions in the remaining five. The model was not trained to handle such edge cases where it failed. These results show the potential of AI to make grading more fair, objective, and efficient.