Distributed Multi-Agent Deep Q-Learning for Load Balancing User Association in Dense Networks

Distributed learning can lead to effective user association with low overhead, but faces significant challenges in incorporating load balancing at all base stations (BS) because of coupling constraints. In this letter, we propose a distributed multi-agent deep Q-learning model for user association to satisfy the load balancing constraint at every BS. Specifically, we design a deep Q-network (DQN) with target Q-network and experience replay buffer at each user as an agent. We also propose a multi-agent matching policy to control the number of users connected to each BS for load balancing. The policy enhances network throughput by implementing a novel updating rule for the preference list at each BS. The proposed multi-agent DQN model operates in a fully distributed manner, where each agent only uses local information and requires no information exchange between agents. Simulation results demonstrate that our proposed algorithm outperforms a conventional distributed load balancing algorithm and approaches a centralized scheme performance, while exhibiting fast convergence and high adaptability to channel changes.

分布式学习能够以较低的开销实现有效的用户关联，但由于耦合约束，在所有基站（BS）实现负载均衡方面面临重大挑战。在本文中，我们提出了一种用于用户关联的分布式多智能体深度Q学习模型，以满足每个基站的负载均衡约束。具体而言，我们在每个用户处设计了一个带有目标Q网络和经验回放缓冲区的深度Q网络（DQN）作为智能体。我们还提出了一种多智能体匹配策略，用于控制连接到每个基站的用户数量以实现负载均衡。该策略通过为每个基站的偏好列表实施一种新的更新规则来提高网络吞吐量。所提出的多智能体DQN模型以完全分布式的方式运行，其中每个智能体仅使用本地信息，并且智能体之间无需信息交换。仿真结果表明，我们提出的算法优于传统的分布式负载均衡算法，并接近集中式方案的性能，同时表现出快速收敛和对信道变化的高适应性。