Medical Surgery Stream Segmentation to Detect and Track Robotic Tools

This paper presents a real-time image segmentation and tracking system that can operate on a continuous stream of endoscopic surgical videos. Such a system can find several valuable applications in the medical field, such as building real-time augmentations to assist with robotic surgeries, training medical residents, and summarizing surgery videos to generate reports having the whole understanding of the procedure. We have formulated a segmentation technique that requires minimal supervision and provides real-time tracking of the objects. The results from the evaluation of our approach indicate that even with minimal annotated data from surgeons, we can achieve good segmentation. This reduces the need for extensive and expensive data collection and annotation processes from robotic surgery. We evaluated our approach on two datasets, EndoVis 2017 and 2018, a dataset from the Robotic Instrument sub-challenge from MICCAI 2017 and 2018. Our results are on par with the state-of-the-art methods on EndoVis-17 and EndoVis-18 for binary segmentation, and in the case of multi-class, we do it with EndoVis-17. The main contribution of our paper is to give the segmentation results in the real-time streaming data