Stereoscopic augmented reality for single camera endoscopy: a virtual study

Yen Yu Wang, Atul Kumar*, Kai Che Liu, Shih Wei Huang, Ching-Chun Huang, Wei Chia Su, Fu Li Hsiao, Wen Nung Lie

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Endoscopic surgery causes less tissue injury compared to open surgical techniques, thus promoting more rapid recuperation and reduced post-operative pain. Endoscopy, however, allows the surgeon to visualise only the anatomical surface of the surgical site, with a relatively narrow field of view. Moreover, the 2D video captured by the conventional endoscope does not provide depth perception of the surgical scene. In this study, these limitations have been addressed with the development of an augmented reality (AR) system with stereoscopic visualisation. A phantom and its 3D CT model were used, respectively, to form the real and virtual parts of the AR. The virtual environment camera pose was tracked using algorithms for image feature detection, feature matching and Perspective-n-Point applied on the endoscopic image and the 3D virtual model-rendered image. The endoscope video frame- and the virtual model-rendered images were superimposed to form the AR composite view. The depth buffer (z-buffer) of the rendering window was further used to make a stereo pair of the AR image. The AR system produced a stereo composite view having well-aligned real and virtual components. The RMS error of the real and virtual image contours registration was 9.6 ± 6.7 mm. Correlation coefficients between the depth map from z-buffer and a depth camera was between 0.60 and 0.96 (p < 0.05). The AR system requires further improvement to be applicable at a higher frame rate of endoscope image acquisition. It also needs to include motion and deformation models when applied to animals or patients.

Original languageEnglish
Pages (from-to)182-191
Number of pages10
JournalComputer Methods in Biomechanics and Biomedical Engineering: Imaging and Visualization
Issue number2
StatePublished - 4 Mar 2018


  • Endoscopic surgery
  • augmented reality
  • stereo endoscopy


Dive into the research topics of 'Stereoscopic augmented reality for single camera endoscopy: a virtual study'. Together they form a unique fingerprint.

Cite this