Background: Recurrent esophageal cancer is associated with dismal prognosis. There is no consensus about the role of surgical treatments in patients with limited recurrences. This study aimed to evaluate the role of surgical resection in patients with resectable recurrences after curative esophagectomy and to identify their prognostic factors. Methods: We retrospectively reviewed patients with recurrent esophageal cancer after curative esophagectomy between 2004 and 2017 and included those with oligo-recurrence that was amenable for surgical intent. The prognostic factors of overall survival (OS) and post-recurrence survival (PRS), as well as the survival impact of surgical resection, were analyzed. Results: Among 654 patients after curative esophagectomies reviewed, 284 (43.4%) had disease recurrences. The recurrences were found resectable in 63 (9.6%) patients, and 30 (4.6%) patients received surgery. The significant prognostic factors of PRS with poor outcome included mediastinum lymph node (LN) recurrence and pathologic T3 stage. In patients with and without surgical resection for recurrence cancer, the 3-year OS rates were 65.6 and 47.6% (p = 0.108), while the 3-year PRS rates were 42.9 and 23.5% (p = 0.100). In the subgroup analysis, surgery for resectable recurrence, compared with non-surgery, could achieve better PRS for patients without any comorbidities (hazard ratio 0.36, 95% CI: 0.14 to 0.94, p = 0.038). Conclusions: Mediastinum LN recurrence or pathologic T3 was associated with worse OS and PRS in patients with oligo-recurrences after curative esophagectomies. No definite survival benefit was noted in patients undergoing surgery for resectable recurrence, except in those without comorbidities.