Model-Based Soft Actor-Critic

Deep reinforcement learning has been successfully developed for many challenging applications. However, collecting new data in actual environment requires a lot of costs which make the agent to learn slowly for high-dimensional states and actions. It is crucial to enhance the sample efficiency and learn with long-term planning. To tackle these issues, this study presents a stochastic agent driven by a new model-based soft actor-critic (MSAC). The dynamics of the environment as well as the reward function are represented by a learnable world model which allows the agent to explore latent representation of environment which conducts stochastic prediction and foresight planning. An off-policy method is proposed by combining with an online learning for world model. The actor, critic and world model are jointly trained to fulfill multi-step foresight imagination. To further enhance the performance, an overshooting scheme is incorporated for long-term planning, and the multi-step rollout is applied for stochastic prediction. The experiments on various tasks with continuous actions show the merit of the proposed MSAC for data efficiency in reinforcement learning.