How (Mis)calibrated is your Federated CLIP and what to do about it?

Abstract

Vision-language models (VLMs) such as CLIP are increasingly adapted across decentralized data silos, yet the reliability of their predictions under federated learning (FL) remains largely unexplored. In this work, we present a systematic study of calibration in federated CLIP under non-IID client distributions. Our experiments reveal that widely used prompt-tuning methods consistently degrade calibration, often yielding substantially higher calibration error despite competitive recognition performance, while explicit training-time calibration regularizers provide only limited improvements. Motivated by these findings, we identify the choice of fine-tuning parameterization as a critical factor governing calibration and conduct a controlled comparison between prompt tuning and five backbone fine-tuning (BFT) strategies: AdaptFormer, LayerNorm, LoRA, VeRA, and DoRA. We find that BFT methods generally offer a more favorable accuracy-calibration trade-off than prompt tuning, although their benefits are not universal. Through extensive analysis, we show that calibration behavior is closely linked to the geometry of federated updates, residual parameterization, and the resulting client and logit drift. Across in-distribution, domain-generalization, and base-to-new evaluation settings, our results establish fine-tuning parameterization as a central design choice for building accurate and reliable federated CLIP models. Codes are available at https://github.com/mainaksingha01/FL2oRA.