One of the main barriers of free space optical (FSO) communication systems is atmospheric turbulence. Various processing techniques at the transmitter, receiver, and transceiver sides are available for addressing this issue; however, they have either high complexity or low performance. Considering this problem, in this study, deep learning (DL) is deployed at the transmitter, receiver, and transceiver sides of an FSO system for constellation shaping, detection, and joint constellation‐shaping detection, respectively. Furthermore, the proposed DL‐based structures are deployed in an FSO‐multi‐input multi‐output (MIMO) system. As the first investigation over DL for the FSO‐MIMO system, different combining schemes including the maximum ratio combiner, equal gain combiner, and the selection combiner are considered. Considering a wide range of atmospheric turbulence, from the weak to the strong regime, the performance of the proposed structures are compared with that of the maximum likelihood (ML) detection. To the best of the authors\\\\\\\' knowledge, the main contributions and novelties of this work include considering transmitter learning in the FSO system, designing low complexity DL structures for FSO system applications, and providing complexity analysis for the proposed DL algorithms. The results indicate that the proposed DL‐based FSO systems achieve the optimum performance with lower complexity compared with the state‐of‐the‐art conventional FSO systems. For instance, the proposed DL‐based detector is almost 2, 3, and 7.5 times faster than the ML detector for modulation orders of 16, 64, and 256, respectively.

Keywords