Constrained latent state modeling: A unifying perspective on representation learning under competing constraints

Abstract

Learning latent representations from complex data is central to modern machine learning, spanning temporal, multimodal, and partially observed systems. In such settings, representations are more naturally understood as latent states capturing underlying system dynamics rather than compressed summaries of observations. Yet current approaches remain fragmented, relying on distinct, often implicit, assumptions about what these states should represent. We argue that this fragmentation reflects a more fundamental limitation: latent representations are typically learned from underconstrained objectives that fail to specify the properties that meaningful latent states should satisfy. As a result, multiple representations may satisfy the same objective, leading to ambiguity in their structure and interpretation. While many underlying principles have been studied in isolation, their interactions have not been explicitly formalized. We propose Constrained Latent State Modeling (CLSM) as a unifying conceptual framework. CLSM characterizes latent states through complementary constraints, including predictive sufficiency, minimality, temporal coherence, observation compatibility, invariance to nuisance factors, and structural constraints, and interprets representation learning as balancing these properties through trade-offs. Revisiting major modeling families through this lens, we show that existing approaches emphasize different subsets of constraints, occupying distinct regions of a common design space. A controlled synthetic benchmark illustrates how different constraint combinations induce distinct latent organizations and Pareto-optimal trade-offs. By shifting the emphasis from architecture-centric to constraint-driven design, CLSM provides a principled framework for analyzing existing methods, guiding new ones, and evaluating latent representations according to their intended properties.