Multi-agent deep reinforcement learning-based RIS-aided UAV communications

Abstract

However, traditional model-based phase-shift optimization is highly sensitive to imperfect CSI and becomes computationally prohibitive for large UPA-based RIS, while existing model-free solutions relying on single-agent DRL struggle with the exponentially growing action space. This paper presents a scalable multi-agent deep Q-network (MADQN)–based RIS controller designed for large-scale UAV–RIS systems under realistic channel dynamics. An end-to-end channel inference architecture is first introduced to mitigate CSI imperfection and reconstruct stable channel representations under UAV mobility. A multi-objective formulation is then developed to jointly optimize sum rate, energy consumption, and control latency, which is transformed into a multi-agent Markov decision process (MMDP) compatible with quantized RIS hardware. Building on this formulation, a dual-agent RIS controller is proposed, in which row and column agents cooperatively determine the quantized phase configuration of a large UPA RIS. Extensive simulations demonstrate that the proposed framework significantly outperforms benchmark schemes, showing acceptable robustness against varying Rician factor SNRs, UAV densities, and RIS sizes. These results confirm that the proposed MADQN-based controller is a promising and practical solution for scalable RIS control in large-scale multi-UAV communication systems.