The neural mechanisms of visual perceptual learning (VPL) remain unclear. Previously we found that activation in the primary visual cortex (V1) increased in the early encoding phase of training, but returned to baseline levels in the later retention phase. To examine neural changes during the retention phase, we measured structural and functional connectivity changes using MRI. After weeks of training on a texture discrimination task, the fractional anisotropy of the inferior longitudinal fasciculus, a major tract connecting visual and anterior areas, was increased, as well as the functional connectivity between V1 and anterior regions mediated by the ILF. These changes were strongly correlated with behavioral performance improvements. These results suggest a two-phase model of VPL in which localized functional changes in V1 in the encoding phase of training are followed by changes in both structural and functional connectivity in ventral visual processing, perhaps leading to the long-term stabilization of VPL.