Scalable Spatial Memory for Scene Rendering and Navigation

Wen Cheng Chen*, Chu Song Chen, Wei Chen Chiu, Min Chun Hu*


研究成果: Conference contribution同行評審


Neural scene representation and rendering methods have shown promise in learning the implicit form of scene structure without supervision. However, the implicit representation learned in most existing methods is non-expandable and cannot be inferred online for novel scenes, which makes the learned representation difficult to be applied across different reinforcement learning (RL) tasks. In this work, we introduce Scene Memory Network (SMN) to achieve online spatial memory construction and expansion for view rendering in novel scenes. SMN models the camera projection and back-projection as spatially aware memory control processes, where the memory values store the information of the partial 3D area, and the memory keys indicate the position of that area. The memory controller can learn the geometry property from observations without the camera’s intrinsic parameters and depth supervision. We further apply the memory constructed by SMN to exploration and navigation tasks. The experimental results reveal the generalization ability of our proposed SMN in large-scale scene synthesis and its potential to improve the performance of spatial RL tasks.

主出版物標題AAAI-23 Technical Tracks 1
編輯Brian Williams, Yiling Chen, Jennifer Neville
發行者AAAI press
出版狀態Published - 27 6月 2023
事件37th AAAI Conference on Artificial Intelligence, AAAI 2023 - Washington, United States
持續時間: 7 2月 202314 2月 2023


名字Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023


Conference37th AAAI Conference on Artificial Intelligence, AAAI 2023
國家/地區United States


深入研究「Scalable Spatial Memory for Scene Rendering and Navigation」主題。共同形成了獨特的指紋。