The present study aimed to determine whether a recently proposed cerebral small vessel disease (CSVD) classification scheme could differentiate the 5-year all-cause mortality in middle-to-old aged asymptomatic CSVD. Stroke-free and non-demented participants recruited from the community-based I-Lan Longitudinal Aging Study underwent baseline brain magnetic resonance imaging (MRI) between 2011 and 2014 and were followed-up between 2018 and 2019. The study population was classified into control (non-CSVD) and CSVD type 1–4 groups based on MRI markers. We determined the association with mortality using Cox regression models, adjusting for the age, sex, and vascular risk factors. A total of 735 participants were included. During a mean follow-up of 5.7 years, 62 (8.4%) died. There were 335 CSVD type 1 (57.9 ± 5.9 years), 249 type 2 (65.6 ± 8.1 years), 52 type 3 (67.8 ± 9.2 years), and 38 type 4 (64.3 ± 9.0 years). Among the four CSVD types, CSVD type 4 individuals had significantly higher all-cause mortality (adjusted hazard ratio = 5.0, 95% confidence interval 1.6–15.3) compared to controls. This novel MRI-based CSVD classification scheme was able to identify individuals at risk of mortality at an asymptomatic, early stage of disease and might be applied for future community-based health research and policy.