Optimum transceiver designs in two-hop amplify-and-forward MIMO relay systems with SIC receivers

We consider joint source/relay precoding in three-node two-hop amplify-and-forward (AF) multiple-input-multiple-output (MIMO) relay systems. In our systems, linear precoders are used at the source and the relay, and the QR successive interference cancelation (SIC) receiver is used at the destination. Our design criterion is to minimize the block error rate (BLER) of the receiver. Since the BLER is a complicated function of the source and relay precoders, and the power constraints are coupled, the optimization problem is difficult to solve. To overcome the difficulty, we first apply the primal decomposition approach, transforming the original optimization to a subproblem and a master problem. In the subproblem, the optimum source precoder can be obtained with the geometric mean decomposition (GMD). In the master problem, however, the optimum relay precoder cannot be straightforwardly obtained. We theoretically prove that the optimum relay precoder exhibits a matrix diagonalization property. Using this property, we can then transform the master problem into a scalar-variable concave optimization problem. A closed-form solution can be derived by the Karuch-Kuhn-Tucker (KKT) conditions. Finally, we extend our method to the two-hop AF MIMO relay system with the minimum mean square error (MMSE) SIC receiver. Assuming a unitary source precoder, we obtain the optimum source and relay precoders in closed form. Simulations show that the proposed transceivers can significantly improve the system performance.