arXiv subjects

Shihab A. Shamma

Publications and source records attributed to Shihab A. Shamma.

8 recordsLinked to original sources

arXiv2026

Integrating Language Models into Listened and Imagined Speech Decoding from MEG

Decoding imagined speech is an important goal for brain-computer interfaces but remains challenging due to weak neural responses, low signal-to-noise ratio, and limited imagined-speech datasets. Language models provide strong contextual cues for text prediction, but how much they can help neural decoding and whether their contribution differs for decoding perceived and imagined speech remains unclear. To investigate this, we use a paired listened-imagined MEG dataset and incorporate language-model information at two stages. First, we train a contrastive neural decoder that aligns MEG representations with acoustic and contextual language representations, improving cross-subject word decoding for both listened and imagined speech. Second, at inference, we introduce a neural-constrained beam-search framework that combines neural evidence with language-model next-word probabilities. We find that imagined-speech decoding benefits more from the language model than listened-speech decoding. For Imagined speech, the best-performing balance between neural and language-model evidence shifts toward the language model, and the gain over neural-only decoding is larger. Together, these results suggest that language priors are most useful when neural evidence is weaker, making them particularly valuable for imagined-speech BCIs.

cs.LG↗

arXiv2026

Relating the Neural Representations of Vocalized, Mimed, and Imagined Speech

We investigated the relationship among neural representations of vocalized, mimed, and imagined speech recorded using publicly available stereotactic EEG recordings. Most prior studies have focused on decoding speech responses within each condition separately. Here, instead, we explore how responses across conditions relate by training linear spectrogram reconstruction models for each condition and evaluate their generalization across conditions. We demonstrate that linear decoders trained on one condition generally transfer successfully to others, implying shared speech representations. This commonality was assessed with stimulus-level discriminability by performing a rank-based analysis demonstrating preservation of stimulus-specific structure in both within- and across-conditions. Finally, we compared linear reconstructions to those from a nonlinear neural network. While both exhibited cross-condition transfer, linear models achieve superior stimulus-level discriminability.

cs.SD↗

arXiv2025

EEG Blink Artifacts Can Identify Read Music in Listening and Imagery

Eye-movement related artifacts including blinks and saccades are significantly larger in amplitude than cortical activity as recorded by scalp electroencephalography (EEG), but are typically discarded in EEG studies focusing on cognitive mechanisms as explained by cortical source activity. Accumulating evidence however indicates that spontaneous eye blinks are not necessarily random, and can be modulated by attention and cognition beyond just physiological necessities. In this exploratory analysis we reanalyze a public EEG dataset of musicians listening to or imagining music (Bach chorales) while simultaneously reading from a sheet of music. We ask whether blink timing in reading music, accompanied by listening or imagery, is sufficient to uniquely identify the music being read from a given score. Intra-subject blink counts and timing are compared across trials using a spike train distance metric (Victor and Purpura, 1997). One-trial-left-out cross-validation is used to identify the music being read with above chance level accuracy (best subject: 56\%, chance: 25\%), where accuracy is seen to vary with subject, condition, and a tunable cost factor for time shifts. Future studies may consider incorporating eye blink contributions to brain decoding, especially in wearables where eye blinks could be easier to record than EEG given their higher amplitudes.

q-bio.NC↗

arXiv2024

Biomimetic Frontend for Differentiable Audio Processing

While models in audio and speech processing are becoming deeper and more end-to-end, they as a consequence need expensive training on large data, and are often brittle. We build on a classical model of human hearing and make it differentiable, so that we can combine traditional explainable biomimetic signal processing approaches with deep-learning frameworks. This allows us to arrive at an expressive and explainable model that is easily trained on modest amounts of data. We apply this model to audio processing tasks, including classification and enhancement. Results show that our differentiable model surpasses black-box approaches in terms of computational efficiency and robustness, even with little training data. We also discuss other potential applications.

cs.SD↗

arXiv2015

Recursive Sparse Point Process Regression with Application to Spectrotemporal Receptive Field Plasticity Analysis

We consider the problem of estimating the sparse time-varying parameter vectors of a point process model in an online fashion, where the observations and inputs respectively consist of binary and continuous time series. We construct a novel objective function by incorporating a forgetting factor mechanism into the point process log-likelihood to enforce adaptivity and employ $\ell_1$-regularization to capture the sparsity. We provide a rigorous analysis of the maximizers of the objective function, which extends the guarantees of compressed sensing to our setting. We construct two recursive filters for online estimation of the parameter vectors based on proximal optimization techniques, as well as a novel filter for recursive computation of statistical confidence regions. Simulation studies reveal that our algorithms outperform several existing point process filters in terms of trackability, goodness-of-fit and mean square error. We finally apply our filtering algorithms to experimentally recorded spiking data from the ferret primary auditory cortex during attentive behavior in a click rate discrimination task. Our analysis provides new insights into the time-course of the spectrotemporal receptive field plasticity of the auditory neurons.

cs.NE↗

arXiv2006

Temporal Symmetry in Primary Auditory Cortex: Implications for Cortical Connectivity

Neurons in primary auditory cortex (AI) in the ferret (Mustela putorius) that are well described by their spectro-temporal response field (STRF), are found also to have a distinctive property that we call temporal symmetry. For temporally symmetric neurons, every temporal cross-section of the STRF (impulse response) is given by the same function of time, except for a scaling and a "Hilbert rotation". This property held in 85% of neurons (123 out of 145) recorded from awake animals, and in 96% of neurons (70 out of 73) recorded from anesthetized animals. This property of temporal symmetry is highly constraining for possible models of functional neural connectivity within and into AI. We find that the simplest models of functional thalamic input, from the ventral Medial Geniculate Body (MGB), into the entry layers of AI are ruled out because they are incompatible with the constraints of the observed temporal symmetry. This is also the case for the simplest models of functional intracortical connectivity. Plausible models that do generate temporal symmetry, both from thalamic and intracortical inputs, are presented. In particular, we propose that two specific characteristics of the thalamo-cortical interface may be responsible. The first is a temporal mismatch between the fast dynamics of the thalamus and the slow responses of the cortex. The second is that all thalamic inputs into a cortical module (or a cluster of cells) must be restricted to one point of entry (or one cell in the cluster). This latter property implies a lack of correlated horizontal interactions across cortical modules during the STRF measurements. The implications of these insights in the auditory system, and comparisons with similar properties in the visual system, are explored.

q-bio.NC↗

arXiv2005

Stimulus-invariant processing and spectrotemporal reverse correlation in primary auditory cortex

The spectrotemporal receptive field (STRF) provides a versatile and integrated, spectral and temporal, functional characterization of single cells in primary auditory cortex (AI). In this paper, we explore the origin of, and relationship between, different ways of measuring and analyzing an STRF. We demonstrate that STRFs measured using a spectrotemporally diverse array of broadband stimuli -- such as dynamic ripples, spectrotemporally white noise, and temporally orthogonal ripple combinations (TORCs) -- are very similar, confirming earlier findings that the STRF is a robust linear descriptor of the cell. We also present a new deterministic analysis framework that employs the Fourier series to describe the spectrotemporal modulations contained in the stimuli and responses. Additional insights into the STRF measurements, including the nature and interpretation of measurement errors, is presented using the Fourier transform, coupled to singular-value decomposition (SVD), and variability analyses including bootstrap. The results promote the utility of the STRF as a core functional descriptor of neurons in AI.

q-bio.NC↗

arXiv1998

Measuring the dynamics of neural responses in primary auditory cortex

We review recent developments in the measurement of the dynamics of the response properties of auditory cortical neurons to broadband sounds, which is closely related to the perception of timbre. The emphasis is on a method that characterizes the spectro-temporal properties of single neurons to dynamic, broadband sounds, akin to the drifting gratings used in vision. The method treats the spectral and temporal aspects of the response on an equal footing.

q-bio.NC↗