Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation

Abstract

The LiDAR 3D object detector that balances accuracy and speed is crucial for achieving real-time perception in autonomous driving. However, many existing LiDAR detection models rely on complex feature transformations, leading to poor real-time performance and high resource consumption, which limits their practical effectiveness. In this work, we propose a Faster LiDAR 3D object detection framework that Adaptively aligns Sparse voxels to enable efficient heterogeneous knowledge Distillation, called FASD. We aim to distill the Transformer's sequence modeling capability into Mamba models, significantly boosting accuracy through knowledge transfer. Specifically, we first design a cross-model knowledge distillation architecture to convey the global contextual understanding capabilities of the Transformer to Mamba. The Transformer-based teacher model employs a scale-adaptive attention mechanism to enhance multi-scale fusion. In contrast, the Mamba-based student model leverages feature alignment through spatial alignment adapters, supervised with latent-space features and span-head logit distributions, leading to improved performance and efficiency. We evaluated FASD on the Waymo and nuScenes datasets, achieving up to a 2x reduction in FLOPs and a 4x reduction in memory consumption, while improving baseline performance by 1-2 percentage points and maintaining high deployment efficiency.