Sexual dimorphism in the brain is known to underpin sex differences in neuropsychological behaviors. The white matter (WM) microstructure appears to be coupled with cognitive performances. However, the issues concerning sex differences in WM remains to be determined. This study used the tract-based spatial statistics on diffusion tensor imaging concurrently with the assessments of Empathizing Quotient (EQ) and Systemizing Quotient (SQ) in forty healthy female and forty male adults. Females exhibited greater fractional anisotropy (FA) in the fronto-occipital fasciculus, body of the corpus callosum, and WM underlying the parahippocampal gyrus. Males exhibited larger FA in the bilateral internal capsule, WM underlying the medial frontal gyrus, fusiform gyrus, hippocampus, insula, postcentral gyrus, frontal and temporal lobe. Interestingly, the interaction analysis of dispositional measures by sex showed that females had a positive correlation between FA of the WM underlying the inferior parietal lobule and superior temporal gyrus and EQ but a negative correlation between FA of the occipital and postcentral gyrus and SQ. Males displayed the opposite effect. The findings indicate a sexual dimorphism of WM microstructure. Divergent correlations of WM microstructure and neuropsychological behaviors between sexes may account for the higher prevalence of autism spectrum disorders in males.