Tobacco smoking has been found associated with lower cardiorespiratory fitness in white and black males; however, few studies have not been conducted to clarify such relationship in Asian males. We performed a cross-sectional study to investigate the association between tobacco smoking status and physical fitness in 3,669 military males, averaged 29.4 years of age, from the cardiorespiratory fitness and hospitalization events in armed forces (CHIEF) study in Taiwan during 2014. There were 1,376 current smokers, and the others were noncurrent smokers. The effective sample size estimated was 1,230 participants, as the margin of error was ±3% at the 99% confidence level. Physical fitness was evaluated by time for a 3000-meter run test (aerobic fitness) and repetitive numbers of 2-minute sit-ups and 2-minute push-ups (anaerobic fitness) where all procedures were standardized by using computerized scoring systems. A multiple linear analysis adjusting for age, service specialty, body mass index, heart rate, alcohol intake, and training frequency was used to determine the relationship. As compared with noncurrent smoking, current smoking was inversely correlated with longer time for a 3000-meter run (β = 15.66 (95% confidence intervals (CI): 10.62, 20.70)) and fewer repetitive numbers of 2-minute sit-ups and 2-minute push-ups (β = -1.53 (95% CI: -2.08, -0.97) and -1.31 (95% CI: -2.12, -0.50), respectively). Our finding reconfirms the concept that tobacco smoking might reduce both aerobic and anaerobic fitness among young Asian males.