The few-mode fiber (FMF) nonlinear effects including Kerr nonlinearity and nonlinear coupling are the most important barriers in FMF-based transmission. The performance degradation due to FMF nonlinearity can be mitigated by properly designing the constellation points. The location and occurrence probability of constellation points can be optimized with geometric constellation shaping (GCS) and probabilistic constellation shaping (PCS), respectively. In this paper, we present end-toend deep learning (EEDL)-based GCS, PCS, and joint geometric-probabilistic constellation Shaping (JGPCS) algorithms for FMF systems. The performance of the proposed algorithms is compared with the uniform distributed quadrature amplitude modulation (QAM) and the well-known Maxwell– Boltzmann distributed QAM, in terms of mutual information (MI). Simulation results show 0.15, 0.19, and 0.22 bits/symbol MI improvement respectively for proposed EEDL-based GCS, PCS, and JGPCS algorithms compared with uniform distributed QAM constellation.

Keywords