CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts

Abstract

Self-supervised learning (SSL) through masked autoencoders (MAEs) has recently attracted great attention for remote sensing (RS) foundation model (FM) development, enabling improved representation learning across diverse sensors and downstream tasks. However, existing RS FMs often either suffer from substantial computational complexity during both training and inference or exhibit limited representational capacity. In addition, their pretraining datasets can contain redundant images, which increases training cost without improving the learned representations. These issues restrict their practical applicability in RS. To address these limitations, we improve the computational efficiency of RS FMs along two axes: i) model efficiency; and ii) training-data efficiency. Model efficiency is achieved by integrating the Soft mixture-of-experts (MoE) mechanism into the FM, which allows modality-specific expert processing alongside shared cross-sensor representation learning while reducing computational complexity at both training and inference time. We apply this adaptation to the Cross-Sensor Masked Autoencoder (CSMAE) model, which serves as our main baseline, resulting in the Cross-Sensor Mixture-of-Experts (CSMoE) model. Training-data efficiency is achieved by a thematic-climatic descriptor-driven sampling strategy, which constructs a reduced training set from a large-scale image archive while retaining its geographic and thematic-climatic diversity, and thus reduces pretraining cost. Extensive experiments on scene classification, semantic segmentation, and content-based image retrieval (CBIR) show that CSMoE remains competitive with state-of-the-art RS FMs while requiring substantially fewer floating-point operations (FLOPs). The associated code for the model and the training set creation, as well as the pretrained model weights, will be available at https://git.tu-berlin.de/rsim/csmoe.