Multiview Representation Learning via Distributed Joint Latent Space Structuring

Abstract

We study distributed multiview representation learning, a problem in which $K$ clients each observe a distinct but possibly statistically correlated view. The clients independently extract local representations from their views, which are then used by a central decoder for joint target estimation. One central difficulty is that, since the clients are not allowed to communicate with each other, they must autonomously decide what to encode. We study this coordination problem from a generalization error perspective. For both classification and regression tasks, we derive novel generalization bounds expressed in terms of the Minimum Description Length (MDL) of the joint latent variables across all views and across both training and test datasets. Our structure-aware bound reveals that statistical correlations among the extracted representations tighten the bound, providing theoretical grounding for the empirically observed benefits of cross-view feature alignment. Perhaps counterintuitively, our findings imply that encoders may benefit from extracting redundant representations. Motivated by these bounds, we introduce a data-dependent Gaussian product mixture prior that can be learned and applied in a fully distributed manner. The joint structure of this multiview prior captures inter-view dependencies that are typically discarded by marginal-only approaches. Comprehensive experiments across multiple datasets, encoder architectures, numbers of views, and distortion settings demonstrate the effectiveness of our proposed approach.