Sensing-assisted Communication Beamforming Based on Multi-Modal Feature Extraction for High-Reliable IoV

This paper introduces a sensing-assisted communication method, which relies on the extraction of multi-modal features. Multi-modal data, e.g. vision, radar, lidar, and position are employed as the input data of the proposed beamforming method. The recognition and beamforming accuracy are therefore improved. Initially, the 3D-Conv model is utilized to extract features from the encoded multimodal data. Subsequently, the generative pre-trained transformer (GPT) is employed to grasp correlations across diverse models and fuse their latent features. These fusion features are used to facilitate beam prediction, thereby approximating the optimal beam index for real-world data. Experimental results based on real-world data validate the effectiveness of our approach, achieving an accuracy of 85%, surpassing traditional single-modal schemes by over 25%.