Meta-hierarchical reinforcement learning based beamforming for near-field multi-user communications

Abstract

Extremely Large Antenna Array Systems (ELAAs) promise to provide ultra-high data rates for multi-user 6G Communications, leveraging massive antenna elements to enhance spatial diversity and spectral efficiency. Unfortunately, considering the propagation of spherical waves in ELAAs, near-field beamforming suffers from substantial performance degradation, diminished spatial focusing precision, considerable computational overhead, and increased system costs. To address these challenges, this paper develops an intelligent hybrid beamforming scheme to empower near-field multi-user THz communications with meta-hierarchical reinforcement learning (MHRL). On the one hand, a novel codebook is designed with Hierarchical Deep Q Network (H-DQN), dramatically releasing the potential of near-field beamforming in ELAAs. To explain the availability of our codebook, the quantization bound is analyzed along with the corresponding computational complexity. On the other hand, we further develop a meta-beamforming framework by integrating our intelligent codebook to meta-reinforcement learning (MRL)-based hybrid beamformer, accelerating the CSI acquisition for multi-user communication. For deep investigation, the rate loss of our scheme is calculated against the ideal continuous beamformer, which reveals that the quantization error has been properly bounded. Simulation results show that the proposed algorithm can efficiently manage the near-field beam with higher beamforming gain, and the proposed beamforming framework exhibits improved spectral efficiency, as well as increased robustness to ELAAs.