Multi-Tier Deep Reinforcement Learning for Non-Terrestrial Networks

To provide global coverage and ubiquitous wireless services, non-terrestrial networks (NTNs) composed of space-tier, air-tier, and ground-tier stations, have been regarded as a key enabling technology toward the sixth generation (6G) networks. Such a multi-tier architecture, however, induces fundamental challenges in optimizing the overall system performance of the NTN in terms of signaling overheads, complexity, and even analytical formulation of the optimization problem. Interestingly, deep reinforcement learning (DRL) is an effective approach to tackle 'complex' optimization problems. When applied to the NTN, however, different DRL methods should be developed for various stations due to different constraints in each tier, which raises fundamental issues involving the interaction, orchestration, and protocol design of multi-tier DRL architecture. In this article, we provide the essential principles for multi-tier DRL for the NTN, and propose a generalized multi-tier architecture to embrace different deployment scenarios of NTNs. A case study of joint optimization of spectrum chunk allocation, trajectory design, and user association for space-tier, air-tier, and ground-tier stations is provided. The simulation results indicate that different trade-offs between the overall throughput, optimization dimension, and computational complexity of stations can be achieved through configuring the proposed architectures.