Title:Hybrid Beamforming Design for RSMA-enabled Near-Field Integrated Sensing and Communications

Abstract:To enable high data rates and sensing resolutions, integrated sensing and communication (ISAC) networks leverage extremely large antenna arrays and high frequencies, extending the Rayleigh distance and making near-field (NF) spherical wave propagation dominant. This unlocks numerous spatial degrees of freedom, raising the challenge of optimizing them for communication and sensing tradeoffs. To this end, we propose a rate-splitting multiple access (RSMA)-based NF-ISAC transmit scheme utilizing hybrid digital-analog antennas. RSMA enhances interference management, while a variable number of dedicated sensing beams adds beamforming flexibility. The objective is to maximize the minimum communication rate while ensuring multi-target sensing performance by jointly optimizing receive filters, analog and digital beamformers, common rate allocation, and the sensing beam count. To address uncertainty in sensing beam allocation, a rank-zero solution reconstruction method demonstrates that dedicated sensing beams are unnecessary for NF multi-target detection. A penalty dual decomposition (PDD)-based double-loop algorithm is introduced, employing weighted minimum mean-squared error (WMMSE) and quadratic transforms to reformulate communication and sensing rates. Simulations reveal that the proposed scheme: 1) Achieves performance comparable to fully digital beamforming with fewer RF chains, (2) Maintains NF multi-target detection without compromising communication rates, and 3) Significantly outperforms space division multiple access (SDMA) and far-field ISAC systems.