Oxygen is known to play an important role in the multiferroicity of rare earth manganites; however, how this role changes with rare earth elements is still not fully understood. To address this question, we have used resonant soft x-ray scattering spectroscopy to study the F-type (0,τ,0) diffraction peak from the antiferromagnetic order in DyMnO3 and TbMnO3. We focus on the measurements at O K edge of these two manganites, supplemented by the results at Mn L2 and Dy M5 edge of DyMnO3. We show that the electronic states of different elements are coupled more strongly in DyMnO3 than in TbMnO3, presumably due to the stronger lattice distortion and the tendency to develop E-type antiferromagnetism in the ferroelectric state that promote the orbital hybridization. We also show that the anomaly in the correlation length of (0,τ,0) peak in DyMnO3 signifies the exchange interaction between Mn and rare earth spins. Our findings reveal the prominent role of oxygen orbitals in the multiferroicity of rare earth manganites and the distinct energetics between them.