Real-time computer vision tasks are emerging in consumer electronics with lightweight computing performance, which are an exquisite design art to balance the computational efficiency and accuracy. In this paper, we present the embedded background subtraction (EBGS) - an optimization algorithm for the entire process to increase computational efficiency and detection accuracy simultaneously. EBGS exploits a simple and efficient Additive Increase Multiplicative Decrease (AIMD) filter to improve the foreground detection accuracy without spending too much time. Moreover, the design combination between the contracted codebook background subtraction (BGS) model and a random model update is proposed to reduce the time consumption. Experiments demonstrate that EBGS can decrease the computing overhead for the three parts of BGS process simultaneously and achieve real-time performance and satisfactory detection accuracy under challenging environments.