Robust Tomlinson-Harashima source and linear relay precoders design in amplify-and-forward MIMO relay systems

Existing transceiver designs in amplify-and-forward (AF) multiple-input-multiple-output (MIMO) relay systems often assume the availability of perfect channel state informations (CSIs). Robust designs for imperfect CSI have less been considered. In this paper, we propose a robust nonlinear transceiver design for the system with a Tomlinson-Harashima precoder (THP), a linear relay precoder, and a minimum-mean-squared-error (MMSE) receiver. Since two precoders and imperfect CSIs are involved, the robust transceiver design is difficult. To overcome the difficulty, we first propose cascading an additional unitary precoder after the THP. The unitary precoder can not only simplify the optimization but also improve the performance of the MMSE receiver. We then adopt the primal decomposition dividing the original optimization problem into a subproblem and a master problem. With our formulation, the subproblem can be solved and the two-precoder problem can be transferred to a single relay precoder problem. The master problem, however, is not solvable. We then propose a lower bound for the objective function and transfer the master problem into a convex optimization problem. A closed-form solution can then be obtained by the Karush-Kuhn-Tucker (KKT) conditions. Simulations show that the proposed transceiver can significantly outperform existing linear transceivers with perfect or imperfect CSIs.