Cerebral small vessel disease (CSVD) is a common finding on brain magnetic resonance imaging (MRI). We previously demonstrated that high blood pressure (BP) and low carotid flow velocity were associated with cerebrovascular disease. However, their associations with brain volume and CSVD remain to be determined. A total of 721 adults (≥ 50 years) from the community-based I-Lan Longitudinal Aging Study were included. Flow velocities at the common (CCA) and internal carotid artery (ICA), including peak systolic velocity (PSV) and end-diastolic velocity (EDV), were measured with Doppler ultrasound. We further detected the presence of CSVD including lacune, microbleed, and white matter hyperintensity (WMH) on brain MRI, which were used to construct a combined CSVD score. General linear regression and logistic regression analysis were exploited to evaluate the association between carotid flow velocity, BP, brain volume, and CSVD. The mean of white matter, gray matter, and WMH volume were 422.2 cm3, 546.9 cm3, and 2.61 cm3, respectively. The proportion of lacune and microbleed were 11.1% and 14.2%, respectively. The CSVD score were negatively associated with gray (r = − 0.121, p <.01) and white matter volume (r = 0.058, p = 0.12), but positively associated with systolic BP (beta = 1.02, p = 0.0017). EDV at common carotid artery associated positively with white matter volume (beta = 1.013, p = 0.0064) and negatively predicted the presence CSVD (odds ratio [OR] = 0.93, p = 0.0023). In the ordinal logistic regression analysis adjusting for age, sex, total intracranial volume education, low-density lipoprotein cholesterol, and fasting glucose, compared with hypertensive subjects with low EDV, normotensive subjects with low, middle, and high EDV had an odds ratio of 0.656 (0.327–1.318), 0.429 (0.261–0.705), and 0.272 (0.147–0.502) for CSVD score, respectively. High SBP and low carotid flow velocities were independently associated with brain volume and CSVD. These associations may be involved in the pathophysiology of cognitive function decline.