Nonlinear distortions introduced by power amplifiers (PAs) pose a major challenge in wireless communication systems by increasing the symbol error rate (SER) and reducing overall reliability. Conventional solutions are often ineffective under strong nonlinear conditions, either due to reliance on accurate PA models or fixed constellation structures. This paper proposes two deep learning (DL) based detection architectures to address these limitations. The first, termed the DL-detector, employs a neural network-based receiver combined with conventional modulation to improve detection under nonlinear impairments. The second, called the End-to-End model, jointly optimizes the transmitter constellation and receiver detection within a unified DL framework. Simulation results under linear, pseudo-linear, and nonlinear PA regimes show that the End-to-End model consistently outperforms both the ML-detector and the DL-detector. In highly nonlinear scenarios, it achieves orders-of-magnitude SER reduction by adaptively shaping the constellation and forming nonlinear decision boundaries aligned with PA characteristics. These results demonstrate that End-to-End learning provides a robust and scalable approach for reliable communications under realistic hardware impairments.

Keywords