Agent-based box manipulation has wide applications in computer animation and robotics. Deep reinforcement learning can be applied to generate animations of agent-based box manipulation. This paper focuses on push-manipulation in an agent-based animation. A policy is learned in a learning session in which an agent receives a reward that is a combination of different types of reward terms. Based on the received reward, the policy is improved gradually. In this paper, we investigate the effects of each reward term in-depth in a framework that is integrated with deep reinforcement learning. We also propose a simple way to produce different animation types. We performed several examples and analyzed our findings in details.