This paper proposes a new prototype filter design method based on the time-frequency constellation for filter bank multicarrier (FBMC) systems in highly doubly dispersive channels. Maximal spectral efficiency, complex orthogonality and a well-localised pulse shaping in time and frequency domains are three criteria in the prototype filter design. Satisfying these criteria simultaneously is a challenging issue in the FBMC system design. In order to meet these criteria as much as possible, we consider an FBMC system in which the complex-valued data symbols are transmitted via a selected time-frequency constellation with maximal spectral efficiency. Based on the selected constellation and channel statistic properties, the prototype filter is designed by maximising a new utility function called signal to weighted interference ratio (SWIR), which is defined according to the channel time and frequency dispersions. Because the SWIR is a nonlinear utility function, its maximisation is not straightforward. In this paper, for prototype filter design, a reliable, robust and fast converging algorithm called evolutionary cooperatively coevolving particle swarm optimisation is used to maximise the SWIR. The FBMC system, which uses the proposed prototype filter design method, can be combined with the MIMO techniques efficiently because of its capability of sending complex-valued symbols. Simulation results show that the performance and bandwidth efficiency of the proposed method are superior to those of other techniques, for both cases of single antenna and multiple antennas, in highly doubly dispersive channels.

Keywords