Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2403.03234

Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling

Yair Schiff·Chia-Hsiang Kao·Aaron Gokaslan·Tri Dao·Albert Gu·Volodymyr Kuleshov
View original source ↗Read source document ↗Export metadata ↓

Abstract

Large-scale sequence modeling has sparked rapid advances that now extend into biology and genomics. However, modeling genomic sequences introduces challenges such as the need to model long-range token interactions, the effects of upstream and downstream regions of the genome, and the reverse complementarity (RC) of DNA. Here, we propose an architecture motivated by these challenges that builds off the long-range Mamba block, and extends it to a BiMamba component that supports bi-directionality, and to a MambaDNA block that additionally supports RC equivariance. We use MambaDNA as the basis of Caduceus, the first family of RC equivariant bi-directional long-range DNA language models, and we introduce pre-training and fine-tuning strategies that yield Caduceus DNA foundation models. Caduceus outperforms previous long-range models on downstream benchmarks; on a challenging long-range variant effect prediction task, Caduceus exceeds the performance of 10x larger models that do not leverage bi-directionality or equivariance.

Explore related subjects

q-bio.GNcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, Volodymyr Kuleshov. 2024-06-05. Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling. https://arxiv.org/abs/2403.03234

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

EMMA: an R/Bioconductor package to automate tracking of metadata in functional enrichment analyses

Summary: Functional enrichment analysis (FEA) is a widely used approach for interpreting high-throughput omics data. However, essential methodological details, such as software versions, analysis parameters, and annotation database releases among others, are often incompletely reported, limiting the reproducibility and transparency of enrichment analyses and complicating the assessment of potentially problematic methodological choices. Here we present EMMA, an R/Bioconductor package that integrates with existing FEA tools and automatically captures provenance metadata, such as annotation metadata, software version, and parameters, during the analysis runtime. Our package provides utilities for accessing and exporting the recorded metadata to facilitate transparent reporting and preserve provenance required for reproducible enrichment analyses. This also enables auditing of the results while remaining compatible with existing Bioconductor workflows. Availability and implementation: EMMA is available on Bioconductor under the MIT license (https: //bioconductor.org/packages/EMMA), with its development version also available on GitHub (https: //github.com/imbeimainz/EMMA).

q-bio.GN↗
arXiv2026

Motif-Vocab: StatisticallyCalibrated Transcription-Factor-Identity Tokenization forGenomic Language Models

Tokenization is a central design choice in genomic language models, yet most deoxyribonucleic acid (DNA) tokenizers use characters, fixed-length k-mers, or frequency-derived subwords without explicitly using prior information about the specificity of DNA-binding regulatory factors. We introduce Motif-Vocab, a biologically informed tokenizer that scans both DNA strands for statistically calibrated motif matches, emits transcription-factor (TF) identity tokens, and applies nucleotide, $k$-mer, or byte-pair encoding (BPE) to unmatched sequence. Motif-specific null distributions put position-weight matrices (PWMs) of different lengths and degeneracy on a common significance scale; deterministic overlap rules make the representation reproducible. In controlled Bidirectional Encoder Representations from Transformers (BERT) pretraining on two billion base pairs, real motif libraries outperform randomized-motif controls on 54 of 55 in-scope downstream tasks. On a motif-disjoint recognition task derived from DART-Eval Task 2, TF-specific tokens improve macro-F1 by 0.040 over a position-matched generic motif token and by 0.033 over a matched no-motif tokenizer (95\% bootstrap confidence interval: 0.027--0.038). Motif tokens also receive stronger attribution and produce larger occlusion effects than shuffled controls. Dense no-motif tokenizers remain strong general-purpose baselines, including a near-tie on the five-task BERT-base panel. Thus, Motif-Vocab is not a universal accuracy replacement; it is a targeted, interpretable inductive bias for motif-sensitive genomic modeling.

q-bio.GN↗
arXiv2026

Revolutionizing Genomics with Reinforcement Learning Techniques

In recent years, Reinforcement Learning (RL) has emerged as a powerful tool for solving a wide range of problems, including decision-making and genomics. The exponential growth of raw genomic data over the past two decades has exceeded the capacity of manual analysis, leading to a growing interest in automatic data analysis and processing. RL algorithms are capable of learning from experience with minimal human supervision, making them well-suited for genomic data analysis and interpretation. One of the key benefits of using RL is the reduced cost associated with collecting labeled training data, which is required for supervised learning. While there have been numerous studies examining the applications of Machine Learning (ML) in genomics, this survey focuses exclusively on the use of RL in various genomics research fields, including gene regulatory networks (GRNs), genome assembly, and sequence alignment. We present a comprehensive technical overview of existing studies on the application of RL in genomics, highlighting the strengths and limitations of these approaches. We then discuss potential research directions that are worthy of future exploration, including the development of more sophisticated reward functions as RL heavily depends on the accuracy of the reward function, the integration of RL with other machine learning techniques, and the application of RL to new and emerging areas in genomics research. Finally, we present our findings and conclude by summarizing the current state of the field and the future outlook for RL in genomics.

q-bio.GN↗