Rate-distortion optimization of H.264/AVC rate control with novel distortion prediction equation

This study proposes a joint rate-distortion optimization for the H.264 rate control algorithm with a novel distortion prediction equation. First, based on the original distortion theorem, we produced a novel, analytically derived equation for the prediction of distortion. This novel distortion predictor avoids linear regression employed in other distortion predictors, and therefore can considerably speed up rate estimation. Then, this work constructed an optimal quantization parameter selection for encoder processing on the new distortion prediction equation and the JVT-G012 rate estimation equation at block level. Simulations demonstrated that the proposed algorithm had achieved an improvement in image quality and a computational saving of at least 34% over two existing algorithms, including the JVT-G012, which was the recommended rate control algorithm in the H.264 reference software JM15.0.