Three dimensional panorama visualization for endoscopic video

Introduction: Endoscopy is widely used in the surgical world. Reduced surgical injury during endoscopic surgery makes the patient’s life easy. However, the endoscopic surgery becomes a challenge for the novice surgeons because of the narrow field of view and the lack of 3D perception in the 2D endoscope image. Such limitations of the endoscope may be addressed with a 3D panorama created with the endoscopic 2D images. Few studies have been reported to create a 3D panorama from endoscopic images. However, to our knowledge, no study has reported a 3D panorama system which uses only a single camera image from the conventional endoscope to make a 3D shape of the organs, and further stitch those 3D shapes to create a 3D panorama of the surgical scene. Such system would enable surgeons to have an extended field of view of the surgical scene with a 3D perception. Method: Images from the endoscopic surgery video were used in this study. A shape from shading (SfS) algorithm was applied to create a 3D shape of the organs in the images. Characteristic feature points were identified on the images using a feature detection algorithm (SURF). The matching feature points in the consecutive images were found using a feature matching algorithm (BRIEF). An iterative closest point (ICP) algorithm was applied to stitch 3D shapes from the consecutive images to synthesize the 3D panorama. Results: The method was applied on 100 consecutive video frames from an endoscopic video of a patient. The root mean square error for the registration of the consecutive image feature points was <4 mm. The 3D panorama from the method helps in visualizing a larger area of the surgical anatomy. An improved version of the method may be applied to a real time video from the endoscope.