Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images

Tai Te Chu, Chia Chun Chang, An Zi Yen, Hen Hsen Huang, Hsin Hsi Chen

研究成果: Conference contribution同行評審

13 引文 斯高帕斯(Scopus)

摘要

With the development of wearable devices, people nowadays record their life experiences much easier than before. Lifelog retrieval becomes an emerging task. Because of the semantic gap between visual data and textual queries, retrieving lifelog images with text queries could be challenging. This paper proposes an interactive lifelog retrieval system that is aimed at retrieving more intuitive and accurate results. Our system is divided into the offline and the online parts. In the offline part, we aim to incorporate original visual and textual concepts from images into our system utilizing pre-trained word embedding. Moreover, we encode the information of relationships between subjects and objects in images by using a pre-trained relation graph generation model. In the online part, We provide an intuitive frontend with various metadata filters, which not only provides users with a convenient interface, but also a mechanism to exploit detail memory recall to users. In this case, users would clearly know the difference between the concepts in the clusters and efficiently browse the retrieved images clusters in a short time.

原文English
主出版物標題LSC 2020 - Proceedings of the 3rd Annual Workshop on the Lifelog Search Challenge
發行者Association for Computing Machinery, Inc
頁面51-55
頁數5
ISBN(電子)9781450371360
DOIs
出版狀態Published - 6 9月 2020
事件3rd Annual Workshop on the Lifelog Search Challenge, LSC 2020 - Dublin, Ireland
持續時間: 8 6月 202011 6月 2020

出版系列

名字LSC 2020 - Proceedings of the 3rd Annual Workshop on the Lifelog Search Challenge

Conference

Conference3rd Annual Workshop on the Lifelog Search Challenge, LSC 2020
國家/地區Ireland
城市Dublin
期間8/06/2011/06/20

指紋

深入研究「Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images」主題。共同形成了獨特的指紋。

引用此