Robust MMSE transceiver design in amplify-and-forward MIMO relay system with Tomlinson-Harashima source precoding

Most nonlinear transceiver designs in amplify-and-forward (AF) multiple-input-multiple-output (MIMO) relay systems assume that instantaneous perfect channel state informations (CSIs) are available. Transceiver design with imperfect CSIs have rarely been addressed. In this paper, we consider an AF MIMO system in which a Tomlinson-Harashima precoder (THP) is used at the source, a linear precoder at the relay, and a minimum-mean-squared-error (MMSE) receiver at the destination. With the imperfect CSIs, we propose a new robust transceiver design method. It is shown that the optimization problem for the design is difficult due to the fact that the objective function is a nonlinear function of the source and relay precoders and yet the constraints are coupled. To overcome the problem, we adopt the primal decomposition technique, decomposing the original optimization into a subproblem and a master problem. To facilitate the derivation of the solution, we propose cascading THP with a unitary precoder and using a lower bound of the objective function. In this way, we can translate the original matrix-valued optimization into the scalar-valued concave optimization. Using the Karush-Kuhn-Tucker (KKT) conditions, we can finally obtain the closed-form solutions for the relay and source precoders. Simulations show that the proposed design is effective against the imperfect CSIs.