Summary Broadband millimeter wave (mmW) systems are a promising pioneer of cellular communication for next generation which is utilizing the hybrid baseband/ analog beamforming structures along with the miniature massive antenna arrays at both sides of the communication link. mmW channel with an available unlicensed spread spectrum is frequency selective because the signal bandwidth can be larger than the coherence bandwidth. Due to the sparse nature of mmW channel, extracting compressive sensing model of the system is preferable. In fact, exploiting the sparse structure will lead to the reduction of the computational complexity, because there is a reduction in the channel training length compared with the conventional methods such as least square estimation. Most of the prior works have considered on‐grid quantized departure/ arrival angles in the input/output antennas to obtain a sparse virtual channel model. However, the sparse angles in the physical channel model are continuous where this continuity indicates a mismatch between the physical angles and the on‐grid angles. Such a mismatch will contribute to unwanted components in the virtual channel model. Given these extra components, the conventional compressive sensing tools are unable to recover the channel. In this paper, we propose two solutions for overcoming the problem caused by off‐grid angle selection. The first is based on the vector shaping, and the second one is based on the sparse total least square concepts. Simulation results demonstrate that the proposed methods both could obtain an adequate channel recovery and are preferable regarding computational complexity concerning the newly developed surrogate method

Keywords