A Novel CNN Based Standalone Detector for Faster-than-Nyquist Signaling

Abstract

This paper presents a novel convolutional neural network (CNN)-based detector for faster-than-Nyquist (FTN) signaling, introducing structured fixed kernel layers with domain-informed masking to effectively mitigate intersymbol interference (ISI). Unlike standard CNN architectures that rely on moving kernels, the proposed approach employs fixed convolutional kernels at predefined positions to explicitly learn ISI patterns at varying distances from the central symbol. To enhance feature extraction, a hierarchical filter allocation strategy is employed, assigning more filters to earlier layers for stronger ISI components and fewer to later layers for weaker components. This structured design improves feature representation, eliminates redundant computations, and enhances detection accuracy while maintaining computational efficiency. Simulation results demonstrate that the proposed detector achieves near-optimal bit error rate (BER) performance, comparable to the BCJR algorithm for the compression factor $τ\geq 0.7$, while offering up to $46\%$ and $84\%$ computational cost reduction over M-BCJR for BPSK and QPSK, respectively. Additional evaluations confirm the method's adaptability to high-order modulations (up to 64-QAM), resilience in quasi-static multipath Rayleigh fading channels, and effectiveness under LDPC-coded FTN transmission, highlighting its robustness and practicality.