We develop self-supervised models and neural architectures that learn transferable representations from large-scale brain recordings.
Large neural datasets offer an opportunity to learn reusable representations of brain activity. We develop spatiotemporal transformers, neural tokenizers, and self-supervised objectives that account for the structure of neural recordings. BaRISTA studies spatial encoding and masking across recording channels, atlas parcels, and brain regions. WiCAT extends shared representation learning to brain imaging across subjects, tasks, and datasets.
Lucine L. Oganesian, Saba Hashemi, Maryam M. Shanechi
BaRISTA learns representations of human intracranial recordings through self-supervised pretraining. Its transformer architecture separates the spatial scales used for encoding and masking, showing that brain-region information can improve downstream decoding while preserving the reconstruction of individual recording channels.
Intracranial recordings have opened a unique opportunity to simultaneously measure activity across multiregional networks in the human brain. Recent works have focused on developing transformer-based neurofoundation models of such recordings that can generalize across subjects and datasets. However, these recordings exhibit highly complex spatiotemporal interactions across diverse spatial scales, from the single-channel scale to the scale of brain regions. As such, there remain critical open questions regarding how best to encode spatial information and how to design self-supervision tasks that enable the learning of brain network patterns and enhance downstream decoding performance using such high-dimensional, multiregional recordings. To allow for exploring these questions, we propose a new spatiotemporal transformer model of multiregional neural activity and a corresponding self-supervised masked latent reconstruction task, designed to enable flexibility in the spatial scale used for token encoding and masking. Applying this model on publicly available multiregional intracranial electrophysiology (iEEG) data, we demonstrate that adjusting the spatial scale for both token encoding and masked reconstruction significantly impacts downstream decoding. Further, we find that spatial encoding at larger scales than channel-level encoding, which is commonly used in existing iEEG transformer models, improves downstream decoding performance. Finally, we demonstrate that our method allows for region-level token encoding while also maintaining accurate channel-level neural reconstruction. Taken together, our modeling framework enables exploration of the spatial scales used for token encoding and masking, reveals their importance towards self-supervised pretraining of neurofoundation models of multiregional human brain activity, and enhances downstream decoding performance.
WiCAT learns shared representations from widefield calcium imaging across subjects. Atlas-aligned tokenization and self-supervised pretraining support transfer across tasks and datasets, including continuous behavior decoding and reconstruction of unseen brain regions in animals excluded from training.
We build dynamical models that reveal how neural populations encode behavior, respond to inputs, and support real-time decoding.
Neural recordings combine activity related to behavior, internal states, and external inputs. Our models separate and prioritize behaviorally relevant dynamics, identify how inputs change neural systems, and enable flexible inference from incomplete observations. This work connects interpretable dynamical models with recurrent neural networks and model-based approaches to neurotechnology.
DPAD uses recurrent neural networks to separate and prioritize behaviorally relevant neural dynamics. It improves neural and behavioral prediction, preserves behavior information in low-dimensional representations, and tests where nonlinearities arise in neural-to-behavior transformations.
Understanding the dynamical transformation of neural activity to behavior requires new capabilities to nonlinearly model, dissociate and prioritize behaviorally relevant neural dynamics and test hypotheses about the origin of nonlinearity. We present dissociative prioritized analysis of dynamics (DPAD), a nonlinear dynamical modeling approach that enables these capabilities with a multisection neural network architecture and training approach. Analyzing cortical spiking and local field potential activity across four movement tasks, we demonstrate five use-cases. DPAD enabled more accurate neural–behavioral prediction. It identified nonlinear dynamical transformations of local field potentials that were more behavior predictive than traditional power features. Further, DPAD achieved behavior-predictive nonlinear neural dimensionality reduction. It enabled hypothesis testing regarding nonlinearities in neural–behavioral transformation, revealing that, in our datasets, nonlinearities could largely be isolated to the mapping from latent cortical dynamics to behavior. Finally, DPAD extended across continuous, intermittently sampled and categorical behaviors. DPAD provides a powerful tool for nonlinear dynamical modeling and investigation of neural–behavioral data.
Hamidreza Abbaspourazad, Eray Erturk, Bijan Pesaran, Maryam M. Shanechi
DFINE combines nonlinear neural manifolds with tractable latent dynamics to enable real-time filtering, offline smoothing and inference with missing observations, while improving neural and behavioral prediction across diverse datasets.
Modelling the spatiotemporal dynamics in the activity of neural populations while also enabling their flexible inference is hindered by the complexity and noisiness of neural observations. Here we show that the lower-dimensional nonlinear latent factors and latent structures can be computationally modelled in a manner that allows for flexible inference causally, non-causally and in the presence of missing neural observations. To enable flexible inference, we developed a neural network that separates the model into jointly trained manifold and dynamic latent factors such that nonlinearity is captured through the manifold factors and the dynamics can be modelled in tractable linear form on this nonlinear manifold. We show that the model, which we named ‘DFINE’ (for ‘dynamical flexible inference for nonlinear embeddings’) achieves flexible inference in simulations of nonlinear dynamics and across neural datasets representing a diversity of brain regions and behaviours. Compared with earlier neural-network models, DFINE enables flexible inference, better predicts neural activity and behaviour, and better captures the latent neural manifold structure. DFINE may advance the development of neurotechnology and investigations in neuroscience.
BRAID incorporates measured external inputs into nonlinear neural-behavioral models, separating intrinsic dynamics from input effects. Prioritized training and a forecasting objective improve learning of behavior-related dynamics and prediction of future activity and behavior.
IPSID jointly models neural activity, behavior and measured inputs to distinguish intrinsic behavior-related dynamics from input-driven activity. It reveals low-dimensional intrinsic dynamics shared across subjects and tasks in the studied motor datasets.
Omid G. Sani, Hamidreza Abbaspourazad, Yan T. Wong, Bijan Pesaran, Maryam M. Shanechi
PSID learns neural dynamical models that prioritize activity related to measured behavior. It reveals lower-dimensional, more behavior-predictive dynamics than behavior-agnostic models, with demonstrations across reach-and-grasp and saccade tasks.
Yuxiao Yang, Shaoyu Qiao, Omid G. Sani, J. Isaac Sedillo, Breonna Ferrentino, Bijan Pesaran, Maryam M. Shanechi
Dynamic input-output models predict how temporally varying electrical stimulation changes activity across brain regions, providing a basis for precise neuromodulation and model-based control.
We learn from neural images and complementary signals, capturing brain dynamics across subjects, modalities, and timescales.
Neural imaging captures rich patterns across space and time. WiCAT learns shared representations of widefield calcium imaging across subjects; SBIND identifies behaviorally relevant dynamics in calcium and functional ultrasound images. In multimodal recordings, MRINE combines signals such as spikes and field potentials with different sampling rates, statistical distributions, and missing observations to support real-time decoding.
Mohammad Hosseini, Eray Erturk, Saba Hashemi, Maryam M. Shanechi
WiCAT learns shared representations from widefield calcium imaging across subjects. Atlas-aligned tokenization and self-supervised pretraining support transfer across tasks and datasets, including continuous behavior decoding and reconstruction of unseen brain regions in animals excluded from training.
Large-scale, multi-subject widefield calcium imaging provides unprecedented access to brain-wide cortical dynamics. However, the high dimensionality, complex spatiotemporal structure, and substantial task-irrelevant activity in widefield recordings have largely restricted modeling efforts to single-session analyses, limiting scalability and generalization. While multi-subject pretrained models have been explored for some neural modalities, multi-subject models for widefield calcium imaging have not yet been demonstrated; further, subject-invariant zero-shot behavior decoding remains elusive for multi-subject models across neural modalities more broadly. As a first step toward foundation modeling of widefield data, we introduce WiCAT, a multi-subject model that leverages self-supervised pretraining to both outperform single-session models and enable zero-shot behavior decoding on unseen subjects. WiCAT introduces an atlas-grounded tokenization scheme without session-specific components and learns globally shared spatiotemporal representations. Across multiple widefield datasets, the pretrained model supports lightweight downstream decoding, transfers across subjects, tasks, and datasets, and outperforms baseline models. Notably, the model also achieves robust zero-shot continuous behavior decoding and left-out brain region reconstruction on unseen subjects.
SBIND models behaviorally relevant dynamics directly from neural image sequences. By capturing local and long-range spatial relationships and separating behavior-related activity from other dynamics, it improves neural and behavioral prediction in widefield calcium and functional ultrasound imaging.
High-dimensional imaging of neural activity, such as widefield calcium and functional ultrasound imaging, provide a rich source of information for understanding the relationship between brain activity and behavior. Accurately modeling neural dynamics in these modalities is crucial for understanding this relationship but is hindered by the high-dimensionality, complex spatiotemporal dependencies, and prevalent behaviorally irrelevant dynamics in these modalities. Existing dynamical models often employ preprocessing steps to obtain low-dimensional representations from neural image modalities. However, this process can discard behaviorally relevant information and miss spatiotemporal structure. We propose SBIND, a novel data-driven deep learning framework to model spatiotemporal dependencies in neural images and disentangle their behaviorally relevant dynamics from other neural dynamics. We validate SBIND on widefield imaging datasets, and show its extension to functional ultrasound imaging, a recent modality whose dynamical modeling has largely remained unexplored. We find that our model effectively identifies both local and long-range spatial dependencies across the brain while also dissociating behaviorally relevant neural dynamics. Doing so, SBIND outperforms existing models in neural-behavioral prediction. Overall, SBIND provides a versatile tool for investigating the neural mechanisms underlying behavior using imaging modalities.
MRINE combines neural modalities with different sampling rates and statistical distributions to decode behavior in real time. Its multiscale dynamical model fuses complementary information from signals such as spikes and field potentials while handling asynchronous observations and missing samples.
Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.
