Markus Marks

Functional magnetic resonance imaging (fMRI) based on the blood-oxygen-level-dependent (BOLD) signal is widely used to study brain activity non-invasively. However, the relationship between the local neural population activity and vascular activity, as measured by BOLD fMRI, remains incompletely understood. Here we show that simultaneous measurement of cellular calcium (Ca ²⁺ ) activity and whole-brain BOLD fMRI in awake mice can reveal spatially specific relationships between neurons and the surrounding vasculature. We introduce an MRI-compatible single-photon microscope that enables recordings from genetically defined neurons at cellular resolution during fMRI. Using this approach, we find that neurons located close to blood vessels often show a negative relationship with the local BOLD signal, whereas neurons farther away display a more variable, often positive relationship. We further demonstrate that local neural activity can be linked to BOLD responses across distributed, connected brain regions. Together, these results provide insight into how vascular organization shapes fMRI signals and establish a powerful tool for bridging cellular and whole-brain measurements of brain function.

Ultra-flexible electrodes, due to their superior biocompatibility, are likely to lead the future of neuroprosthetics. However, identifying the precise positions of implanted high-density ultra-flexible electrodes in the brain for accurately assigning neural signals to specific structures remains a major challenge. To address this, we developed magnetic resonance identification (MRID)-tags. Each ultra-flexible electrode bundle carries an MRID-tag with unique barcode patterns visible in MRI (MRI-barcodes) for identification of the bundle. Individual bars in MRI-barcodes allow an accurate 3D reconstruction of the ultra-flexible electrode bundle’s trajectory in the brain and determine the anatomical positions of individual electrodes. We generate the MRI-barcodes by patterning superparamagnetic iron-oxide nanoparticles into electrode fibers (10 µm ² ) with dot-matrix nanoparticle coating technique. We chronically tested MRID-tagged ultra-flexible electrodes in vivo in the dorsal hippocampus of freely-moving rats, where distinct electrophysiological landmarks validated our electrode localization results. We were able to localize individual electrodes with a mean accuracy of 95 μm. MRID-tagged ultra-flexible electrodes demonstrated high long-term recording stability with mean single-unit signal-to-noise ratios as high as 20.

Cells are a fundamental unit of biological organization, and identifying them in imaging data—cell segmentation—is a critical task for various cellular imaging experiments. Although deep learning methods have led to substantial progress on this problem, most models are specialist models that work well for specific domains but cannot be applied across domains or scale well with large amounts of data. Here we present CellSAM, a universal model for cell segmentation that generalizes across diverse cellular imaging data. CellSAM builds on top of the Segment Anything Model (SAM) by developing a prompt engineering approach for mask generation. We train an object detector, CellFinder, to automatically detect cells and prompt SAM to generate segmentations. We show that this approach allows a single model to achieve human-level performance for segmenting images of mammalian cells, yeast and bacteria collected across various imaging modalities. We show that CellSAM has strong zero-shot performance and can be improved with a few examples via few-shot learning. Additionally, we demonstrate how CellSAM can be applied across diverse bioimage analysis workflows. A deployed version of CellSAM is available at https://cellsam.deepcell.org/ .

Ultra-flexible electrodes, due to their superior biocompatibility, are likely to lead the future of neuroprosthetics. However, identifying the locations of high-density ultra-flexible electrodes in the brain for accurately assigning neural signals to specific structures remains a major challenge. To address this, we developed magnetic resonance identification (MRID) -tags. Each ultra-flexible electrode bundle carries an MRID-tag with unique barcode patterns visible in MRI (MRI-barcodes) for identification of the bundle and the anatomical positions of individual electrodes. We generate the MRI-barcodes by patterning superparamagnetic iron-oxide nanoparticles into electrode fibers (10µm ² ) with dot-matrix nanoparticle coating technique. We chronically tested MRID-tagged ultra-flexible electrodes in vivo in the dorsal hippocampus of freely-moving rats, where distinct electrophysiological landmarks validated our electrode localization results. We were able to localize individual electrodes with a mean accuracy of 83 µm. MRID-tagged ultra-flexible electrodes demonstrated months-long recording stability with mean single-unit signal-to-noise ratios as high as 20.

Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. The model is forced to learn about the data’s inherent structure or context by solving a pretext task. With SSL, models can learn from abundant and cheap unlabeled data, significantly reducing the cost of training models where labels are expensive or inaccessible. In Computer Vision, SSL is widely used as pre-training followed by a downstream task, such as supervised transfer, few-shot learning on smaller labeled data sets, and/or unsupervised clustering. Unfortunately, it is infeasible to evaluate SSL methods on all possible downstream tasks and objectively measure the quality of the learned representation. Instead, SSL methods are evaluated using in-domain evaluation protocols, such as fine-tuning, linear probing, and k-nearest neighbors (kNN). However, it is not well understood how well these evaluation protocols estimate the representation quality of a pre-trained model for different downstream tasks under different conditions, such as dataset, metric, and model architecture. In this work, we study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types. Our study includes eleven common image datasets and 26 models that were pre-trained with different SSL methods or have different model backbones. We find that in-domain linear/kNN probing protocols are, on average, the best general predictors for out-of-domain performance. We further investigate the importance of batch normalization for the various protocols and evaluate how robust correlations are for different kinds of dataset domain shifts. In addition, we challenge assumptions about the relationship between discriminative and generative self-supervised methods, finding that most of their performance differences can be explained by changes to model backbones.

Summary: Artificial activation of anatomically localized, genetically defined hypothalamic neuron populations is known to trigger distinct innate behaviors, suggesting a hypothalamic nucleus-centered organization of behavior control. To assess whether the encoding of behavior is similarly anatomically confined, we performed simultaneous neuron recordings across twenty hypothalamic regions in freely moving animals. Here we show that distinct but anatomically distributed neuron ensembles encode the social and fear behavior classes, primarily through mixed selectivity. While behavior class-encoding ensembles were spatially distributed, individual ensembles exhibited strong localization bias. Encoding models identified that behavior actions, but not motion-related variables, explained a large fraction of hypothalamic neuron activity variance. These results identify unexpected complexity in the hypothalamic encoding of instincts and provide a foundation for understanding the role of distributed neural representations in the expression of behaviors driven by hardwired circuits.

Cells are the fundamental unit of biological organization, and identifying them in imaging data - cell segmentation - is a critical task for various cellular imaging experiments. While deep learning methods have led to substantial progress on this problem, models that have seen wide use are specialist models that work well for specific domains. Methods that have learned the general notion of "what is a cell" and can identify them across different domains of cellular imaging data have proven elusive. In this work, we present CellSAM, a foundation model for cell segmentation that generalizes across diverse cellular imaging data. CellSAM builds on top of the Segment Anything Model (SAM) by developing a prompt engineering approach to mask generation. We train an object detector, CellFinder, to automatically detect cells and prompt SAM to generate segmentations. We show that this approach allows a single model to achieve state-of-the-art performance for segmenting images of mammalian cells (in tissues and cell culture), yeast, and bacteria collected with various imaging modalities. To enable accessibility, we integrate CellSAM into DeepCell Label to further accelerate human-in-the-loop labeling strategies for cellular imaging data. A deployed version of CellSAM is available at https://label-dev.deepcell.org/.

Blood Oxygen Level-Dependent (BOLD) functional Magnetic Resonance Imaging (fMRI) allows for non-invasive, indirect recordings of neural activity across the whole brain in both humans and animals. However, the relationship between the local neural population activity and the vascular activity is not completely understood. To investigate this relationship, we present a novel MRI compatible single-photon microscope capable of measuring cellular resolution Ca ²⁺ activity of genetically defined neurons during whole-brain BOLD fMRI in awake behaving mice. Using this combined imaging approach, we found a difference in activity patterns between cells which was dependent on their location with respect to the vasculature. Notably, neurons near the vasculature showed pronounced negative activity during contralateral whisker movements at 3 Hz. In a second proof of concept experiment, we demonstrate the potential of recording both local neural activities, like those in the barrel field (SSp-bfd), and BOLD fMRI readings from interlinked brain regions. In sum, the presented technological advancement paves the way for studies examining the interplay between local brain circuits and overarching brain functions. In addition, the new approach enhances our understanding of the vascular BOLD fMRI signal, providing insights into the determinants of local neurovascular functions and the brain’s organizational framework across various scales.

Determining the cultural propensities or cultural behaviours of a species during foraging entails an investigation of underlying drivers and motivations. In this article, we propose a multicomponent approach involving behaviour, ecology, and physiology to accelerate the study of cultural propensities in the wild. We propose as the first component the use of field experiments that simulate natural contexts, such as foraging behaviours and tool use opportunities, to explore social learning and cultural tendencies in a variety of species. To further accelerate this component, we discuss and advocate for the use of modern machine learning video analysis tools. In conjunction, we examine non‐invasive methods to measure ecological influences on foraging such as phenology, fruit availability, dietary intake; and physiological influences such as stress, protein balance, energetics, and metabolism. We feature non‐invasive urine sampling to investigate urea, creatinine, ketone bodies, the thyroid hormone triiodothyronine (T3), cortisol and connecting peptides of insulin. To conclude, we highlight the benefits of combining ecological and physiological conditions with behavioural field experiments. This can be done across wild species, and provides the framework needed to test ecological hypotheses related to cultural behaviour.

Quantification of behaviours of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyse the behaviour of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behaviour—even in complex environments directly from raw video frames—that requires no intervention after initial human supervision. Our behavioural classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation and classification of complex behaviour, outperforming the state of the art. SIPEC successfully recognizes multiple behaviours of freely moving individual mice as well as socially interacting non-human primates in three dimensions, using data only from simple mono-vision cameras in home-cage set-ups.