Nonlinear Transceiver Designs for Full-Duplex MIMO Relay Systems

This paper investigates nonlinear transceiver design for full-duplex multiple-input multiple-output (FD-MIMO) relay systems. A dual-hop amplify-and-forward relaying protocol is considered. At the destination, nonlinear successive-interference-cancellation (SIC) is used for signal detection. The goal is to find the source and relay precoders such that the symbol-vector error rate (SVER) can be minimized. Due to the loop interference (LI), optimizing the relay precoder in FD systems is much more involved. In this paper, we propose novel designs to solve this problem. Starting from the QR-SIC receiver, we theoretically show that the relay precoder can be solved with a closed-form expression even when the system incurs LI. Then, we consider the system with a minimum mean-squared-error SIC receiver, where the relay precoder design entails a different problem formulation and introduces new challenges. We propose a novel iterative method, with closed-form solutions in each iteration, to solve this problem. Simulations show that our designs can significantly improve the SVER performance for FD-MIMO relay systems.