Timbre-enhanced Multi-modal Music Style Transfer with Domain Balance Loss

Style transfer of the polyphonic music recordings has always been a challenging task due to the difficulty of learning representations for both domain invariant (i.e. content) and domain-variant (i.e. style) features of the music. Although there exists prior works which employ the Multi-modal Unsupervised Image-to-Image Translation (MUNIT) framework to perform the music style transfer in an unsupervised manner and successfully provide the promising results, the gap between the transferred music recordings and the real ones is still noticeable. In order to reduce such gap, we propose and experiment several techniques for improving the transferred results, including the domain balanced loss, up-sampling, content discriminator, recycle loss, and the data scaling. We conduct extensive experiments on the tasks of bilateral style transfer among four different genres, namely: piano solo, guitar solo, string quartet, and chiptune. In evaluation, an objective testing scheme is proposed to investigate the pros and cons of all our proposed techniques, while we also design a subjective testing method for making comparison among different approaches and show that our proposed method is able to provide superior performance with respect to the prior works.