Swallowing is a complex movement consisting of the sequential and orderly activation of the swallowing muscles. Neuroimaging evidence has revealed a complex cortical and subcortical representation of voluntary swallowing. The repetitive saliva swallowing test (RSST) is a convenient and simple method for assessing swallowing performance in older people. It remains unclear whether individual differences in swallowing performance are associated with variations in structural brain signatures. We aimed to investigate the association between swallowing efficiency (SWE, measured by the RSST) and gray matter volume (GMV) in healthy older adults. Forty healthy older adults (52–82 years old, 28 female) underwent T1-weighted magnetic resonance imaging (MRI) and clinical assessments of SWE, stimulated/unstimulated salivary flow rate, masticatory cycle, and walking speed. GMV was quantified using a voxel-based morphometry (VBM) approach based on the MRI data. SWE was significantly negatively correlated with age. The association between SWE and hand grip strength, but not the other clinical metrics, was statistically significant. The GMV of the left posterior cerebellum (from cerebellum crus to lobule VIII) was significantly positively correlated with SWE, as evidenced by the results of whole-brain and cerebellum-specific VBM analyses. SWE was significantly positively correlated with the cerebellar volume in the region-of-interest analyses based on automated segmentation. In healthy older adults, swallowing efficiency was positively correlated with cerebellar GMV. The findings suggested that in older people, structural variations of the brain may play a key role in individual differences in swallowing performance.