Current imaging tools are insufficiently sensitive to the early diagnosis of esophageal squamous cell carcinoma (ESCC). The application of polarization-sensitive optical coherence tomography (PS-OCT) to detect tumor-stroma interaction is an interesting issue in cancer diagnosis. In this translational study, we found that en-face PS-OCT effectively characterizes protruding, flat, and depressive type ESCC regardless of animal or human specimens. In addition, the tumor contour and margin could also be drawn and determined on a broad en-face view. The determined tumor margin could be in the proximity of 2 mm to the actual tumor margin, which was proved directly using histology.