Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2508.21243

Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification

Aditya Makineni·Baocheng Geng·Qing Tian
View original source ↗Read source document ↗Export metadata ↓

Abstract

Transformers and State-Space Models (SSMs) have advanced audio classification by modeling spectrograms as sequences of patches. However, existing models such as the Audio Spectrogram Transformer (AST) and Audio Mamba (AuM) adopt square patching from computer vision, which disrupts continuous frequency patterns and produces an excessive number of patches, slowing training, and increasing computation. We propose Full-Frequency Temporal Patching (FFTP), a patching strategy that better matches the time-frequency asymmetry of spectrograms by spanning full frequency bands with localized temporal context, preserving harmonic structure, and significantly reducing patch count and computation. We also introduce SpecMask, a patch-aligned spectrogram augmentation that combines full-frequency and localized time-frequency masks under a fixed masking budget, enhancing temporal robustness while preserving spectral continuity. When applied on both AST and AuM, our patching method with SpecMask improves mAP by up to +6.76 on AudioSet-18k and accuracy by up to +8.46 on SpeechCommandsV2, while reducing computation by up to 83.26%, demonstrating both performance and efficiency gains.

Explore related subjects

cs.SDcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Aditya Makineni, Baocheng Geng, Qing Tian. 2025-08-28. Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification. https://arxiv.org/abs/2508.21243

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Retain-Free Machine Unlearning for Speech Emotion Recognition

Speech Emotion Recognition (SER) infers a speaker's emotional state from speech and is increasingly deployed in human-computer interaction, education, and healthcare. Because speech also carries sensitive personal information, speakers may ask that some of their recordings be deleted, which requires removing the influence of those samples from an already trained SER model. Most machine unlearning methods can meet this request only with access to the remaining training data alongside the samples to be forgotten; this is impractical when the remaining data cannot be redistributed or has itself been deleted, and it adds storage and computation as the data grows. To this end, we propose a retain-free unlearning method that updates a pre-trained SER model using only the forget set. Our key idea is to synthesise adversarial samples from the forget set as a surrogate for the unavailable remaining data, and to constrain each parameter update by its estimated importance so that forgetting does not erase general knowledge. The experiments over several emotional-speech corpora and self-supervised backbones show that our method drives forget-set performance down to near chance while retaining much of the model's utility on the remaining and unseen-speaker data, narrowing the gap to methods that rely on the remaining set.

cs.SD↗
arXiv2026

SHINE: Sequential Hierarchical Integration Network for EEG and MEG

How natural speech is represented in the brain constitutes a major challenge for cognitive neuroscience. Reconstructing the speech envelope and Mel spectrogram from EEG and MEG provides a time-resolved way to study its temporal and spectral structure. Speech-related neural activity spans sensors and temporal scales; extracting these representations while adapting the use of context to each acoustic target is a central problem in speech reconstruction. We propose SHINE, a Sequential Hierarchical Integration Network for EEG and MEG. A residual sensor adapter unifies input dimensions, intermediate dilated-block states retain temporal depth, and a target- and time-dependent gate fuses local hierarchical and attention-enhanced context predictions. Across two EEG and two MEG datasets, SHINE has the highest mean envelope and mean-Mel Pearson correlations among nine local baseline implementations on all eight dataset-metric combinations. SHINE also placed second in the speech-detection Extended Track of the NeurIPS 2025 PNPL Competition. Code will be released at https://github.com/xuxiran/SHINE.

cs.SD↗
arXiv2026

Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech

Text-to-speech that reads raw text has no lexicon: a rare word is read as guessed, and a native Japanese listener accepts a word only if its reading and pitch accent are both right. A known remedy installs a reading-and-accent channel into a released model, but it needs many recordings. This paper removes the recordings: the frozen backbone reads a sentence containing a common word it already says correctly, and that output serves as the teacher for the same sentence with the word replaced by an annotated reading with its pitch accent. Screened raters judged the tag right on 0.80 to 0.93 of unseen difficult words on four backbones spanning autoregressive, diffusion, and encoder-decoder synthesis; plain kana, which cannot express an accent, got 0.38 to 0.60. On words needing no edit, naturalness is non-inferior on one backbone; on the other three, listeners prefer the unedited rendition by 0.19 to 0.26.

cs.SD↗