Unsupervised sound localization via iterative contrastive learning

Yan Bo Lin, Hung Yu Tseng, Hsin Ying Lee, Yen Yu Lin*, Ming Hsuan Yang


研究成果: Article同行評審

3 引文 斯高帕斯(Scopus)


Sound localization aims to find the source of the audio signal in the visual scene. However, it is labor-intensive to annotate the correlations between the signals sampled from the audio and visual modalities, thus making it difficult to supervise the learning of a machine for this task. In this work, we propose an iterative contrastive learning framework that requires no data annotations. At each iteration, the proposed method takes the (1) localization results in images predicted in the previous iteration, and (2) semantic relationships inferred from the audio signals as the pseudo-labels. We then use the pseudo-labels to learn the correlation between the visual and audio signals sampled from the same video (intra-frame sampling) as well as the association between those extracted across videos (inter-frame relation). Our iterative strategy gradually encourages the localization of the sounding objects and reduces the correlation between the non-sounding regions and the reference audio. Quantitative and qualitative experimental results demonstrate that the proposed framework performs favorably against existing unsupervised and weakly-supervised methods on the sound localization task.

期刊Computer Vision and Image Understanding
出版狀態Published - 1月 2023


深入研究「Unsupervised sound localization via iterative contrastive learning」主題。共同形成了獨特的指紋。