arXiv subjects

David Keyes

Publications and source records attributed to David Keyes.

At least 19 recordsLinked to original sources

arXiv2026

Foundation Model-Assisted Full Waveform Inversion

Full waveform inversion (FWI) can recover high-resolution subsurface velocity models. Conventional waveform-difference objectives, however, are vulnerable to cycle skipping when the starting model is inaccurate. We introduce an FWI objective that compares features produced from modeled and observed seismic traces by SeisLM, a pretrained seismic foundation model. The SeisLM encoder remains frozen during inversion, and the feature discrepancy is differentiated with respect to the modeled traces to construct an adjoint source compatible with the standard adjoint-state framework. We also test a scheduled hybrid loss that combines the SeisLM feature loss with the conventional $L_2$ objective. Time-shift diagnostics show that the loss computed from features produced by the pretrained encoder has a broader and smoother basin around the correct alignment than either the waveform $L_2$ objective or the feature loss obtained from an encoder with the same architecture and randomly initialized parameters. In the Marmousi experiment, the SeisLM and hybrid objectives produce similar improvements during early-stage inversion and provide useful models for subsequent reflection-based $L_2$ refinement. In the 2D Overthrust experiment, which begins from a laterally invariant linear-gradient model, the SeisLM feature-loss workflow outperforms the conventional and hybrid workflows, indicating that introducing the $L_2$ contribution too early can reintroduce cycle-skipping sensitivity. In the 3D Overthrust experiment, conventional $L_2$ inversion stalls near the initial linear gradient, whereas the SeisLM feature loss guides the inversion toward a background model from which $L_2$ refinement recovers the principal structures. These results support using features produced by pretrained seismic networks to define early-stage FWI objectives rather than complete replacements for waveform-domain misfits.

physics.geo-ph↗

arXiv2026

RAP: KV-Cache Compression via RoPE-Aligned Pruning

Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v projection weights to reduce the output KV dimensions. However, modern LLMs apply Rotary Position Embedding (RoPE) after the QK projections, which rotates feature dimensions in pairs. Therefore, removing individual channels breaks these pairs, corrupting RoPE's positional semantics and rendering the pruned model unusable. We propose RoPE-Aligned Pruning (RAP), which constrains the pruning granularity to RoPE-aligned pairs rather than individual channels: removing whole pairs to keep the rotation intact. Our evaluation across Llama, Mistral, and Qwen models from 3B to 14B shows that RAP preserves accuracy at 30% KV compression (retain ratio \r{ho} = 0.7), far outperforms RoPE-blind channel pruning, stays near the strongest low-rank method at lower attention cost, and composes with orthogonal methods such as quantization.

cs.LG↗

arXiv2025

A Deep Learning-based time shift objective function for Full Waveform Inversion

Full Waveform Inversion (FWI) is a powerful technique for estimating high-resolution subsurface velocity models by minimizing the discrepancy between modeled and observed seismic data. However, the oscillatory nature of seismic waveforms makes point-wise discrepancy measures highly prone to cycle-skipping, especially when the initial velocity model is inadequate. To address this challenge, various alternative misfit functions have been proposed in the literature, each with unique strengths and limitations. Dynamic Time Warping (DTW) is a popular technique in signal processing for aligning time series using dynamic programming. While a differentiable variant of DTW has been recently proposed, its use in FWI is hindered by high-frequency artifacts in the adjoint source and the substantial computational cost of gradient evaluations. In this study, we propose a neural network-based approach to learn the time shifts that align two time series in a supervised manner. The trained network is then utilized to compare traces from observed and modeled seismic data, offering a stable and computationally efficient alternative to DTW. Furthermore, the inherent differentiability of neural networks via backpropagation enables seamless integration into the FWI framework as a misfit function. We validate this approach on two synthetic datasets, namely the Marmousi model and the Chevron blind test dataset, demonstrating in both cases a similar convergence behavior to that of SoftDWT whilst drastically reducing the computational time of the adjoint source calculation.

physics.geo-ph↗

arXiv2025

Linear Complexity $\mathcal{H}^2$ Direct Solver for Fine-Grained Parallel Architectures

We present factorization and solution phases for a new linear complexity direct solver designed for concurrent batch operations on fine-grained parallel architectures, for matrices amenable to hierarchical representation. We focus on the strong-admissibility-based $\mathcal{H}^2$ format, where strong recursive skeletonization factorization compresses remote interactions. We build upon previous implementations of $\mathcal{H}^2$ matrix construction for efficient factorization and solution algorithm design, which are illustrated graphically in stepwise detail. The algorithms are ``blackbox'' in the sense that the only inputs are the matrix and right-hand side, without analytical or geometrical information about the origin of the system. We demonstrate linear complexity scaling in both time and memory on four representative families of dense matrices up to one million in size. Parallel scaling up to 16 threads is enabled by a multi-level matrix graph coloring and avoidance of dynamic memory allocations thanks to prefix-sum memory management. An experimental backward error analysis is included. We break down the timings of different phases, identify phases that are memory-bandwidth limited, and discuss alternatives for phases that may be sensitive to the trend to employ lower precisions for performance.

cs.DC↗

arXiv2025

Synthetic Geology: Structural Geology Meets Deep Learning

Reconstructing the structural geology and mineral composition of the first few kilometers of the Earth's subsurface from sparse or indirect surface observations remains a long-standing challenge with critical applications in mineral exploration, geohazard assessment, and geotechnical engineering. This inherently ill-posed problem is often addressed by classical geophysical inversion methods, which typically yield a single maximum-likelihood model that fails to capture the full range of plausible geology. The adoption of modern deep learning methods has been limited by the lack of large 3D training datasets. We address this gap with \textit{StructuralGeo}, a geological simulation engine that mimics eons of tectonic, magmatic, and sedimentary processes to generate a virtually limitless supply of realistic synthetic 3D lithological models. Using this dataset, we train both unconditional and conditional generative flow-matching models with a 3D attention U-Net architecture. The resulting foundation model can reconstruct multiple plausible 3D scenarios from surface topography and sparse borehole data, depicting structures such as layers, faults, folds, and dikes. By sampling many reconstructions from the same observations, we introduce a probabilistic framework for estimating the size and extent of subsurface features. While the realism of the output is bounded by the fidelity of the training data to true geology, this combination of simulation and generative AI functions offers a flexible prior for probabilistic modeling, regional fine-tuning, and use as an AI-based regularizer in traditional geophysical inversion workflows.

cs.CV↗

arXiv2025

GPU-Accelerated Parallel Selected Inversion for Structured Matrices Using sTiles

Selected inversion is essential for applications such as Bayesian inference, electronic structure calculations, and inverse covariance estimation, where computing only specific elements of large sparse matrix inverses significantly reduces computational and memory overhead. We present an efficient implementation of a two-phase parallel algorithm for computing selected elements of the inverse of a sparse symmetric matrix A, which can be expressed as A = LL^T through sparse Cholesky factorization. Our approach leverages a tile-based structure, focusing on selected dense tiles to optimize computational efficiency and parallelism. While the focus is on arrowhead matrices, the method can be extended to handle general structured matrices. Performance evaluations on a dual-socket 26-core Intel Xeon CPU server demonstrate that sTiles outperforms state-of-the-art direct solvers such as Panua-PARDISO, achieving up to 13X speedup on large-scale structured matrices. Additionally, our GPU implementation using an NVIDIA A100 GPU demonstrates substantial acceleration over its CPU counterpart, achieving up to 5X speedup for large, high-bandwidth matrices with high computational intensity. These results underscore the robustness and versatility of sTiles, validating its effectiveness across various densities and problem configurations.

cs.PF↗

arXiv2025

H2-MG: A multigrid method for hierarchical rank structured matrices

This paper presents a new fast iterative solver for large systems involving kernel matrices. Advantageous aspects of H2 matrix approximations and the multigrid method are hybridized to create the H2-MG algorithm. This combination provides the time and memory efficiency of H2 operator representation along with the rapid convergence of a multilevel method. We describe how H2-MG works, show its linear complexity, and demonstrate its effectiveness on two standard kernels and on a single-layer potential boundary element discretization with complex geometry. The current zoo of H2 solvers, which includes a wide variety of iterative and direct solvers, so far lacks a method that exploits multiple levels of resolution, commonly referred to in the iterative methods literature as ``multigrid'' from its origins in a hierarchy of grids used to discretize differential equations. This makes H2-MG a valuable addition to the collection of H2 solvers. The algorithm has potential for advancing various fields that require the solution of large, dense, symmetric positive definite matrices.

math.NA↗

arXiv2025

sTiles: An Accelerated Computational Framework for Sparse Factorizations of Structured Matrices

This paper introduces sTiles, a GPU-accelerated framework for factorizing sparse structured symmetric matrices. By leveraging tile algorithms for fine-grained computations, sTiles uses a structure-aware task execution flow to handle challenging arrowhead sparse matrices with variable bandwidths, common in scientific and engineering fields. It minimizes fill-in during Cholesky factorization using permutation techniques and employs a static scheduler to manage tasks on shared-memory systems with GPU accelerators. sTiles balances tile size and parallelism, where larger tiles enhance algorithmic intensity but increase floating-point operations and memory usage, while parallelism is constrained by the arrowhead structure. To expose more parallelism, a left-looking Cholesky variant breaks sequential dependencies in trailing submatrix updates via tree reductions. Evaluations show sTiles achieves speedups of up to 8.41X, 9.34X, 5.07X, and 11.08X compared to CHOLMOD, SymPACK, MUMPS, and PARDISO, respectively, and a 5X speedup compared to a 32-core AMD EPYC CPU on an NVIDIA A100 GPU. Our generic software framework imports well-established concepts from dense matrix computations but they all require customizations in their deployments on hybrid architectures to best handle factorizations of sparse matrices with arrowhead structures.

cs.PF↗

arXiv2024

Reciprocity-aware adaptive tile low-rank factorization for large-scale 3D multidimensional deconvolution

Low-rank regularization is an effective technique for addressing ill-posed inverse problems when the unknown variable exhibits low-rank characteristics. However, global low-rank assumptions do not always hold for seismic wavefields; in many practical situations, local low-rank features are instead more commonly observed. To leverage this insight, we propose partitioning the unknown variable into tiles, each represented via low-rank factorization. We apply this framework to regularize multidimensional deconvolution in the frequency domain, considering two key factors. First, the unknown variable, referred to as the Green's function, must maintain symmetry according to the reciprocity principle of wave propagation. To ensure symmetry within the tile-based low-rank framework, diagonal tiles are formulated as the product of a low-rank factor and its transpose if numerically rank-deficient. Otherwise, they are represented by preconditioned dense forms. Symmetry in off-diagonal elements is achieved by parameterizing sub-diagonal tiles as the product of two distinct low-rank factors, with the corresponding super-diagonal tiles set as their transposes. Second, the rank of the Green's function varies with frequency; in other words, the Green's function has different ranks at different frequencies. To determine the numerical rank and optimal tile size for each frequency, we first solve the multidimensional deconvolution problem using a benchmark solver. Based on these results, we estimate the optimal tile size and numerical rank for our proposed solver.

physics.geo-ph↗

arXiv2024

Constructing artificial life and materials scientists with accelerated AI using Deep AndersoNN

Deep AndersoNN accelerates AI by exploiting the continuum limit as the number of explicit layers in a neural network approaches infinity and can be taken as a single implicit layer, known as a deep equilibrium model. Solving for deep equilibrium model parameters reduces to a nonlinear fixed point iteration problem, enabling the use of vector-to-vector iterative solvers and windowing techniques, such as Anderson extrapolation, for accelerating convergence to the fixed point deep equilibrium. Here we show that Deep AndersoNN achieves up to an order of magnitude of speed-up in training and inference. The method is demonstrated on density functional theory results for industrial applications by constructing artificial life and materials `scientists' capable of classifying drugs as strongly or weakly polar, metal-organic frameworks by pore size, and crystalline materials as metals, semiconductors, and insulators, using graph images of node-neighbor representations transformed from atom-bond networks. Results exhibit accuracy up to 98\% and showcase synergy between Deep AndersoNN and machine learning capabilities of modern computing architectures, such as GPUs, for accelerated computational life and materials science by quickly identifying structure-property relationships. This paves the way for saving up to 90\% of compute required for AI, reducing its carbon footprint by up to 60 gigatons per year by 2030, and scaling above memory limits of explicit neural networks in life and materials science, and beyond.

cs.LG↗

arXiv2024

Multidimensional deconvolution with shared bases

We address the estimation of seismic wavefields by means of Multidimensional Deconvolution (MDD) for various redatuming applications. While offering more accuracy than conventional correlation-based redatuming methods, MDD faces challenges due to the ill-posed nature of the underlying inverse problem and the requirement to handle large, dense, complex-valued matrices. These obstacles have long limited the adoption of MDD in the geophysical community. Recent interest in this technology has spurred the development of new strategies to enhance the robustness of the inversion process and reduce its computational overhead. We present a novel approach that extends the concept of block low-rank approximations, usually applied to linear operators, to simultaneously compress the operator, right-hand side, and unknowns. This technique greatly alleviates the data-heavy nature of MDD. Moreover, since in 3d applications the matrices do not lend themselves to global low rank approximations, we introduce a novel H2-like approximation. We aim to streamline MDD implementations, fostering efficiency and controlling accuracy in wavefield reconstruction. This innovation holds potential for broader applications in the geophysical domain, possibly revolutionizing the analysis of multi-dimensional seismic datasets.

math.NA↗

arXiv2024

PETScML: Second-order solvers for training regression problems in Scientific Machine Learning

In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis, by means of deep-learning techniques, of data produced by computational science and engineering applications. At the core of these methods is the supervised training algorithm to learn the neural network realization, a highly non-convex optimization problem that is usually solved using stochastic gradient methods. However, distinct from deep-learning practice, scientific machine-learning training problems feature a much larger volume of smooth data and better characterizations of the empirical risk functions, which make them suited for conventional solvers for unconstrained optimization. We introduce a lightweight software framework built on top of the Portable and Extensible Toolkit for Scientific computation to bridge the gap between deep-learning software and conventional solvers for unconstrained minimization. We empirically demonstrate the superior efficacy of a trust region method based on the Gauss-Newton approximation of the Hessian in improving the generalization errors arising from regression tasks when learning surrogate models for a wide range of scientific machine-learning techniques and test cases. All the conventional second-order solvers tested, including L-BFGS and inexact Newton with line-search, compare favorably, either in terms of cost or accuracy, with the adaptive first-order methods used to validate the surrogate models.

cs.LG↗

arXiv2023

A reciprocity-aware, low-rank regularization for multidimensional deconvolution

This paper presents a novel factorization-based, low-rank regularization method for solving multidimensional deconvolution problems in the frequency domain. In this approach, each frequency component of the unknown wavefield is represented as a complex-valued square matrix and approximated using the product of one rectangular matrix and its transpose. The benefit of such a parametrization is two-fold: first, the size of the unknown matrix is greatly reduced compared to that of the original wavefield of interest (and halved compared to conventional factorization-based, low-rank approximations); second, the retrieved wavefield is implicitly guaranteed to comply with the reciprocity principle, as expected from theory. We further show that the proposed objective function can be successfully optimized using the accelerated proximal gradient algorithm and discuss a robust strategy to define the initial guess of the solution. Numerical examples on synthetic and field data demonstrate the effectiveness of the proposed method in compressing the retrieved Green's function while preserving its accuracy.

physics.geo-ph↗

arXiv2021

Space-Fractional Diffusion with Variable Order and Diffusivity: Discretization and Direct Solution Strategies

We consider the multidimensional space-fractional diffusion equations with spatially varying diffusivity and fractional order. Significant computational challenges are encountered when solving these equations due both to the kernel singularity in the fractional integral operator and to the resulting dense discretized operators, which quickly become prohibitively expensive to handle because of their memory and arithmetic complexities. In this work, we present a singularity-aware discretization scheme that regularizes the singular integrals through a singularity subtraction technique adapted to the spatial variability of diffusivity and fractional order. This regularization strategy is conveniently formulated as a sparse matrix correction that is added to the dense operator, and is applicable to different formulations of fractional diffusion equations. We also present a block low rank representation to handle the dense matrix representations, by exploiting the ability to approximate blocks of the resulting formally dense matrix by low rank factorizations. A Cholesky factorization solver operates directly on this representation using the low rank blocks as its atomic computational tiles, and achieves high performance on multicore hardware. Numerical results show that the singularity treatment is robust, substantially reduces discretization errors, and attains the first-order convergence rate allowed by the regularity of the solutions. They also show that considerable savings are obtained in storage ($O(N^{1.5})$) and computational cost ($O(N^2)$) compared to dense factorizations. This translates to orders-of-magnitude savings in memory and time on multi-dimensional problems, and shows that the proposed methods offer practical tools for tackling large nonlocal fractional diffusion simulations.

math.NA↗

arXiv2021

H2OPUS-TLR: High Performance Tile Low Rank Symmetric Factorizations using Adaptive Randomized Approximation

Tile low rank representations of dense matrices partition them into blocks of roughly uniform size, where each off-diagonal tile is compressed and stored as its own low rank factorization. They offer an attractive representation for many data-sparse dense operators that appear in practical applications, where substantial compression and a much smaller memory footprint can be achieved. TLR matrices are a compromise between the simplicity of a regular perfectly-strided data structure and the optimal complexity of the unbalanced trees of hierarchically low rank matrices, and provide a convenient performance-tuning parameter through their tile size that can be proportioned to take into account the cache size where the tiles reside in the memory hierarchy. There are currently no high-performance algorithms that can generate Cholesky and $LDL^T$ factorizations, particularly on GPUs. The difficulties in achieving high performance when factoring TLR matrices come from the expensive compression operations that must be performed during the factorization process and the adaptive rank distribution of the tiles that causes an irregular work pattern for the processing cores. In this work, we develop a dynamic batching operation and combine it with batched adaptive randomized approximations to achieve high performance both on GPUs and CPUs. Our implementation attains over 1.2 TFLOP/s in double precision on the V100 GPU, and is limited by the performance of batched GEMM operations. The Cholesky factorization of covariance matrix of size $N = 131K$ arising in spatial statistics can be factored to an accuracy $\epsilon=10^{-2}$ in just a few seconds. We believe the proposed GEMM-centric algorithm allows it to be readily ported to newer hardware such as the tensor cores that are optimized for small GEMM operations.

cs.DC↗

arXiv2020

Nonuniform 3D finite difference elastic wave simulation on staggered grids

We present an approach to simulate the 3D isotropic elastic wave propagation using nonuniform finite difference discretization on staggered grids. Specifically, we consider simulation domains composed of layers of uniform grids with different grid spacings, separated by nonconforming interfaces. We demonstrate that this layer-wise finite difference discretization has the potential to significantly reduce the simulation cost, compared to its fully uniform counterpart. Stability of such a discretization is achieved by using specially designed difference operators, which are variants of the standard difference operators with adaptations near boundaries or interfaces, and penalty terms, which are appended to the discretized wave system to weakly impose boundary or interface conditions. Combined with specially designed interpolation operators, the discretized wave system is shown to preserve the energy conserving property of the continuous elastic wave equation, and $\textit{a fortiori}$ ensure the stability of the simulation. Numerical examples are presented to demonstrate the efficacy of the proposed simulation approach.

math.NA↗

arXiv2020

Explicit coupling of acoustic and elastic wave propagation in finite difference simulations

We present a mechanism to explicitly couple the finite-difference discretizations of 2D acoustic and isotropic elastic wave systems that are separated by straight interfaces. Such coupled simulations allow the application of the elastic model to geological regions that are of special interest for seismic exploration studies (e.g., the areas surrounding salt bodies), while with the computationally more tractable acoustic model still being applied in the background regions. Specifically, the acoustic wave system is expressed in terms of velocity and pressure while the elastic wave system is expressed in terms of velocity and stress. Both systems are posed in first-order forms and discretized on staggered grids. Special variants of the standard finite-difference operators, namely, operators that possess the summation-by-parts property, are used for the approximation of spatial derivatives. Penalty terms, which are also referred to as the simultaneous approximation terms, are designed to weakly impose the elastic-acoustic interface conditions in the finite-difference discretizations and couple the elastic and acoustic wave simulations together. With the presented mechanism, we are able to perform the coupled elastic-acoustic wave simulations stably and accurately. Moreover, it is shown that the energy-conserving property in the continuous systems can be preserved in the discretization with carefully designed penalty terms.

physics.comp-ph↗

arXiv2020

Hierarchical Matrix Approximations of Hessians Arising in Inverse Problems Governed by PDEs

Hessian operators arising in inverse problems governed by partial differential equations (PDEs) play a critical role in delivering efficient, dimension-independent convergence for both Newton solution of deterministic inverse problems, as well as Markov chain Monte Carlo sampling of posteriors in the Bayesian setting. These methods require the ability to repeatedly perform such operations on the Hessian as multiplication with arbitrary vectors, solving linear systems, inversion, and (inverse) square root. Unfortunately, the Hessian is a (formally) dense, implicitly-defined operator that is intractable to form explicitly for practical inverse problems, requiring as many PDE solves as inversion parameters. Low rank approximations are effective when the data contain limited information about the parameters, but become prohibitive as the data become more informative. However, the Hessians for many inverse problems arising in practical applications can be well approximated by matrices that have hierarchically low rank structure. Hierarchical matrix representations promise to overcome the high complexity of dense representations and provide effective data structures and matrix operations that have only log-linear complexity. In this work, we describe algorithms for constructing and updating hierarchical matrix approximations of Hessians, and illustrate them on a number of representative inverse problems involving time-dependent diffusion, advection-dominated transport, frequency domain acoustic wave propagation, and low frequency Maxwell equations, demonstrating up to an order of magnitude speedup compared to globally low rank approximations.

math.NA↗