arXiv subjects

Tom Jacobs

Publications and source records attributed to Tom Jacobs.

16 recordsLinked to original sources

arXiv2026

Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics are approximated by a Wasserstein gradient flow that converges to a unique stationary measure. We characterize the structure of this stationary measure and the predictor it represents. We show that, despite the network being infinitely overparameterized, the learned predictor admits an effectively finite representation: the input weights and biases align along finitely many directions, leading to an effective width collapse. In particular, the solution function is continuous piecewise affine, with affine regions determined by the cells of a finite hyperplane arrangement. The number of learned directions, and hence hyperplanes, is bounded above by $2\mathcal{P}-1$, where $\mathcal{P}$ denotes the number of linear dichotomies realizable on the training inputs. We further establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data. Consequently, the complexity of the learned predictor is governed by the combinatorial geometry of the training data.

cs.LG↗

arXiv2026

SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.

cs.LG↗

arXiv2026

HORST: Composing Optimizer Geometries for Sparse Transformer Training

Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit $L_{\infty}$ bias favoring stability, yet, sparsity requires an $L_1$ bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometry in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing $L_1$ sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently and significantly outperforms AdamW baselines across all sparsity levels, with large gains at higher sparsity.

cs.LG↗

arXiv2026

Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning

We study the max-margin solutions reached by mirror flow in deep neural networks with homogeneous activation functions. Extending classical results on gradient flow, we derive a novel balance equation for mirror flow from convex duality, enabling a characterization of the horizon function governing the induced margin. We further establish max-margin characterizations together with convergence rates and norm growth estimates. Finally, we support our theory through experiments on synthetic datasets and standard vision tasks. Concretely, we show that: (1) distinct non-homogeneous mirror maps can induce the same max-margin solution; (2) convergence can be extremely slow, including exponentially slow regimes; and (3) although all considered mirror maps exhibit feature learning, they can produce markedly different representations, ranging from sparse to dense neuron activations. Together, these results provide a unified perspective on sparse and dense feature learning in homogeneous neural networks, highlighting how mirror maps shape both optimization dynamics and the geometry of the learned classifiers.

cs.LG↗

arXiv2026

Never Saddle for Reparameterized Steepest Descent as Mirror Flow

How does the choice of optimization algorithm shape a model's ability to learn features? To address this question for steepest descent methods --including sign descent, which is closely related to Adam --we introduce steepest mirror flows as a unifying theoretical framework. This framework reveals how optimization geometry governs learning dynamics, implicit bias, and sparsity and it provides two explanations for why Adam and AdamW often outperform SGD in fine-tuning. Focusing on diagonal linear networks and deep diagonal linear reparameterizations (a simplified proxy for attention), we show that steeper descent facilitates both saddle-point escape and feature learning. In contrast, gradient descent requires unrealistically large learning rates to escape saddles, an uncommon regime in fine-tuning. Empirically, we confirm that saddle-point escape is a central challenge in fine-tuning. Furthermore, we demonstrate that decoupled weight decay, as in AdamW, stabilizes feature learning by enforcing novel balance equations. Together, these results highlight two mechanisms how steepest descent can aid modern optimization.

cs.LG↗

arXiv2025

The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis

Sparse neural networks promise efficiency, yet training them effectively remains a fundamental challenge. Despite advances in pruning methods that create sparse architectures, understanding why some sparse structures are better trainable than others with the same level of sparsity remains poorly understood. Aiming to develop a systematic approach to this fundamental problem, we propose a novel theoretical framework based on the theory of graph limits, particularly graphons, that characterizes sparse neural networks in the infinite-width regime. Our key insight is that connectivity patterns of sparse neural networks induced by pruning methods converge to specific graphons as networks' width tends to infinity, which encodes implicit structural biases of different pruning methods. We postulate the Graphon Limit Hypothesis and provide empirical evidence to support it. Leveraging this graphon representation, we derive a Graphon Neural Tangent Kernel (Graphon NTK) to study the training dynamics of sparse networks in the infinite width limit. Graphon NTK provides a general framework for the theoretical analysis of sparse networks. We empirically show that the spectral analysis of Graphon NTK correlates with observed training dynamics of sparse networks, explaining the varying convergence behaviours of different pruning methods. Our framework provides theoretical insights into the impact of connectivity patterns on the trainability of various sparse network architectures.

cs.LG↗

arXiv2025

Pay Attention to Small Weights

Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, this criterion is gradient-free -- the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.

cs.LG↗

arXiv2025

Hyperbolic Aware Minimization: Implicit Bias for Sparsity

Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise overparameterization promotes sparsity, but also yields a small inverse Riemannian metric near zero, slowing down parameter movement and impeding meaningful parameter sign flips. To overcome this obstacle, we propose Hyperbolic Aware Minimization (HAM), which alternates a standard optimizer step with a lightweight hyperbolic mirror step. The mirror step incurs less compute and memory than pointwise overparameterization, reproduces its beneficial hyperbolic geometry for feature learning, and mitigates the small-inverse-metric bottleneck. Our characterization of the implicit bias in the context of underdetermined linear regression provides insights into the mechanism how HAM consistently increases performance --even in the case of dense training, as we demonstrate in experiments with standard vision benchmarks. HAM is especially effective in combination with different sparsification methods, advancing the state of the art.

cs.LG↗

arXiv2025

Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch

The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.

cs.LG↗

arXiv2025

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?

Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understanding their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incorporate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encompasses a broad class of problems, including sparse coding, matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regularization and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.

cs.LG↗

arXiv2024

Mask in the Mirror: Implicit Sparsification

Continuous sparsification strategies are among the most effective methods for reducing the inference costs and memory demands of large-scale neural networks. A key factor in their success is the implicit $L_1$ regularization induced by jointly learning both mask and weight variables, which has been shown experimentally to outperform explicit $L_1$ regularization. We provide a theoretical explanation for this observation by analyzing the learning dynamics, revealing that early continuous sparsification is governed by an implicit $L_2$ regularization that gradually transitions to an $L_1$ penalty over time. Leveraging this insight, we propose a method to dynamically control the strength of this implicit bias. Through an extension of the mirror flow framework, we establish convergence and optimality guarantees in the context of underdetermined linear regression. Our theoretical findings may be of independent interest, as we demonstrate how to enter the rich regime and show that the implicit bias can be controlled via a time-dependent Bregman potential. To validate these insights, we introduce PILoT, a continuous sparsification approach with novel initialization and dynamic regularization, which consistently outperforms baselines in standard experiments.

cs.LG↗

arXiv2023

Giant Outer Transiting Exoplanet Mass (GOT 'EM) Survey: III. Recovery and Confirmation of a Temperate, Mildly Eccentric, Single-Transit Jupiter Orbiting TOI-2010

Large-scale exoplanet surveys like the TESS mission are powerful tools for discovering large numbers of exoplanet candidates. Single-transit events are commonplace within the resulting candidate list due to the unavoidable limitation of observing baseline. These single-transit planets often remain unverified due to their unknown orbital period and consequent difficulty in scheduling follow up observations. In some cases, radial velocity (RV) follow up can constrain the period enough to enable a future targeted transit detection. We present the confirmation of one such planet: TOI-2010 b. Nearly three years of RV coverage determined the period to a level where a broad window search could be undertaken with the Near-Earth Object Surveillance Satellite (NEOSSat), detecting an additional transit. An additional detection in a much later TESS sector solidified our final parameter estimation. We find TOI-2010 b to be a Jovian planet ($M_P = 1.29 \ M_{\rm Jup}$, $R_P = 1.05 \ R_{\rm Jup}$) on a mildly eccentric orbit ($e = 0.21$) with a period of $P = 141.83403$ days. Assuming a simple model with no albedo and perfect heat redistribution, the equilibrium temperature ranges from about 360 K to 450 K from apoastron to periastron. Its wide orbit and bright host star ($V=9.85$) make TOI-2010 b a valuable test-bed for future low-insolation atmospheric analysis.

astro-ph.EP↗

arXiv2023

Kepler's Last Planet Discoveries: Two New Planets and One Single-Transit Candidate from K2 Campaign 19

The Kepler space telescope was responsible for the discovery of over 2,700 confirmed exoplanets, more than half of the total number of exoplanets known today. These discoveries took place during both Kepler's primary mission, when it spent 4 years staring at the same part of the sky, and its extended K2 mission, when a mechanical failure forced it to observe different parts of the sky along the ecliptic. At the very end of the mission, when Kepler was exhausting the last of its fuel reserves, it collected a short set of observations known as K2 Campaign 19. So far, no planets have been discovered in this dataset because it only yielded about a week of high-quality data. Here, we report some of the last planet discoveries made by Kepler in the Campaign 19 dataset. We conducted a visual search of the week of high-quality Campaign 19 data and identified three possible planet transits. Each planet candidate was originally identified with only one recorded transit, from which we were able to estimate the planets' radii and estimate the semimajor axes and orbital periods. Analysis of lower-quality data collected after low fuel pressure caused the telescope's pointing precision to suffer revealed additional transits for two of these candidates, allowing us to statistically validate them as genuine exoplanets. We also tentatively confirm the transits of one planet with TESS. These discoveries demonstrate Kepler's exoplanet detection power, even when it was literally running on fumes.

astro-ph.EP↗

arXiv2022

Planetesimals Around Stars with TESS (PAST): II. An M Dwarf "Dipper" Star with a Long-Lived Disk in the TESS Continuous Viewing Zone

Studies of T Tauri disks inform planet formation theory; observations of variability due to occultation by circumstellar dust are a useful probe of unresolved, planet-forming inner disks, especially around faint M dwarf stars. We report observations of 2M0632, an M dwarf member of the Carina young moving group that was observed by TESS over two one-year intervals. The combined light curve contains >300 dimming events, each lasting a few hours, and as deep as 40% (0.55 magnitudes). These stochastic events are correlated with a distinct, stable 1.86-day periodic signal that could be stellar rotation. Concurrent ground-based, multi-band photometry show reddening consistent with ISM-like dust. The star's excess emission in the infrared and emission lines in optical and infrared spectra, reveal a T Tauri-like accretion disk around the star. We confirm membership of 2M0632 in the Carina group by a Bayesian analysis of its Galactic space motion and position. We combine stellar evolution models with Gaia photometry and constraints on Teff, luminosity, and the absence of detectable lithium in the photosphere to constrain the age of the group and 2M0632 to 40-60 Myr, consistent with earlier estimates. 2M0632 joins a handful of long-lived disks which challenge the canon that disk lifetimes are <10 Myr. All known examples surround M dwarfs, suggesting that lower X-ray/UV irradiation and slower photoevaporation by these stars can dramatically affect disk evolution. The multi-planet systems spawned by long-lived disks probably experienced significant orbital damping and migration into close-in, resonant orbits, and perhaps represented by the TRAPPIST-1 system.

astro-ph.EP↗

arXiv2015

Kepler Eclipsing Binary Stars. VII. The Catalog of Eclipsing Binaries Found in the Entire Kepler Data-Set

The primary Kepler Mission provided nearly continuous monitoring of ~200,000 objects with unprecedented photometric precision. We present the final catalog of eclipsing binary systems within the 105 square degree Kepler field of view. This release incorporates the full extent of the data from the primary mission (Q0-Q17 Data Release). As a result, new systems have been added, additional false positives have been removed, ephemerides and principal parameters have been recomputed, classifications have been revised to rely on analytical models, and eclipse timing variations have been computed for each system. We identify several classes of systems including those that exhibit tertiary eclipse events, systems that show clear evidence of additional bodies, heartbeat systems, systems with changing eclipse depths, and systems exhibiting only one eclipse event over the duration of the mission. We have updated the period and galactic latitude distribution diagrams and included a catalog completeness evaluation. The total number of identified eclipsing and ellipsoidal binary systems in the Kepler field of view has increased to 2878, 1.3% of all observed Kepler targets. An online version of this catalog with downloadable content and visualization tools is maintained at http://keplerEBs.villanova.edu.

astro-ph.SR↗

arXiv2013

Planet Hunters. VI: An Independent Characterization of KOI-351 and Several Long Period Planet Candidates from the Kepler Archival Data

We report the discovery of 14 new transiting planet candidates in the Kepler field from the Planet Hunters citizen science program. None of these candidates overlapped with Kepler Objects of Interest (KOIs) at the time of submission. We report the discovery of one more addition to the six planet candidate system around KOI-351, making it the only seven planet candidate system from Kepler. Additionally, KOI-351 bears some resemblance to our own solar system, with the inner five planets ranging from Earth to mini-Neptune radii and the outer planets being gas giants; however, this system is very compact, with all seven planet candidates orbiting $\lesssim 1$ AU from their host star. A Hill stability test and an orbital integration of the system shows that the system is stable. Furthermore, we significantly add to the population of long period transiting planets; periods range from 124-904 days, eight of them more than one Earth year long. Seven of these 14 candidates reside in their host star's habitable zone.

astro-ph.EP↗