Context Window Failures in Relational Foundation Models

Denis Oliveira Correa; Francisco Galuppo Azevedo

Home/Archive/arXiv

arXiv · 2609.00460

Context Window Failures in Relational Foundation Models

Denis Oliveira Correa·Francisco Galuppo Azevedo

View original source ↗Read source document ↗Export metadata ↓

Abstract

Recent Relational Deep Learning architectures have been proposed as foundation models for multi-table relational data, yet they impose constrained neighborhood budgets that force row truncation when an entity has many related records. We introduce Animus, a synthetic financial dataset in which predicting customer income requires aggregating up to tens of thousands of transactions. On the raw representation, three recently proposed models (RT, Griffin, RelGT) achieve $R^2 \le 0.18$; a single, routine, temporal pre-aggregation step recovers $R^2$ up to $0.65$. This questions whether current relational foundation models are ready for high-cardinality real-world data.

Explore related subjects

cs.LG

Keep this discovery

BibTeX RIS

Denis Oliveira Correa, Francisco Galuppo Azevedo. 2026-08-31. Context Window Failures in Relational Foundation Models. https://arxiv.org/abs/2609.00460

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.LG subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

Higher Structures in Deep Learning

We provide an expository introduction on the importance of higher-arity tensor operations to deep learning. Then, we conduct a novel empirical investigation of higher-arity phenomenon in trained neural networks, introduce a hypergraphical generalization of the multilayer perceptron, and explore connections to evolutionary algorithms. We conclude with a discussion of promising directions for future research.

cs.LG↗

arXiv2026

Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

We prove a non-asymptotic central limit theorem for vector-valued martingale differences using Stein's method, and use Poisson's equation to extend the result to functions of Markov Chains. We then show that these results can be applied to establish a non-asymptotic central limit theorem for Temporal Difference (TD) learning with averaging.

math.PR↗

arXiv2026

One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Context

We extend recent work establishing an equivalence between one-layer transformers and nearest-neighbor classifiers in the binary setting to the multiclass case. By leveraging the simplex encoding, we show that one-layer transformers with an argmax classification head behave identically to a one-nearest-neighbor classifier in the multiclass setting. This closes a gap left by prior work, whose multiclass result relied on a non-standard rounding-based approach rather than the typical argmax head used in practice.

cs.LG↗