Laura Leal-Taixé
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Laura Leal-Taixé has a total of 109 co-authors. Here are the top co-authors by number of shared publications:
- Aljosa Osep
Technical University of Munich Germany
10 shared publications - Patrick Dendorfer
Technical University of Munich
7 shared publications - Daniel Cremers
Technical University of Munich
5 shared publications - Qunjie Zhou
NVIDIA, Santa Clara, USA
5 shared publications - Bastian Leibe
RWTH Aachen University
4 shared publications - Ismail Elezi
Dynamic Vision and Learning Group, Technical University of Munich, Munich, Germany
4 shared publications
Latest Publications
- Open Access
- 30 July 2026
- Published by Springer Nature in Scientific Data
- Vol. 13 (1) , 1105
Human neuron activity during an 83-minute movie from 2,286 neurons and 29 patients
Despite a growing trend towards more naturalistic experiments, few single-unit datasets collected during dynamic and naturalistic stimuli have been released, and none using a full-length movie. Here, we present SUMMER (Single Unit activity during a Movie in the human Medial temporal lobe via Electrophysiological Recordings), a dataset containing recordings from 2,286 neurons from the human amygdala, hippocampus, entorhinal cortex, parahippocampal cortex, and neighboring structures during the complete presentation of the commercial film 500 Days of Summer to 29 intracranially implanted patients. We provide a rich set of frame-wise annotations spanning the entirety of the movie’s 83-minute runtime, which systematically label the most salient narrative and visual elements, from characters and locations to camera cuts and main character speech. This Neurodata Without Borders-formatted dataset contains the spike times and corresponding mean waveforms from all recorded neurons, along with demographic information and electrode localizations. For technical validation, we provide spike-sorting metrics and demonstrate the tuning of individual neurons to movie features. We additionally provide decoding results from a machine learning-based pipeline for predicting movie features from neuronal population activity. To facilitate immediate use, we offer the dataset in a machine learning-ready format alongside a codebase demonstrating both neural feature and movie feature prediction tasks. This dataset offers a valuable foundation for exploring how the human brain processes semantic content, particularly in real-world contexts involving dynamic and naturalistic stimuli.
- 30 July 2026
- Published by Springer Nature in Scientific Data
- Vol. 13 (1) , 1105
Open AccessHuman neuron activity during an 83-minute movie from 2,286 neurons and 29 patients
Despite a growing trend towards more naturalistic experiments, few single-unit datasets collected during dynamic and naturalistic stimuli have been released, and none using a full-length movie. Here, we present SUMMER (Single Unit activity during a Movie in the human Medial temporal lobe via Electrophysiological Recordings), a dataset containing recordings from 2,286 neurons from the human amygdala, hippocampus, entorhinal cortex, parahippocampal cortex, and neighboring structures during the complete presentation of the commercial film 500 Days of Summer to 29 intracranially implanted patients. We provide a rich set of frame-wise annotations spanning the entirety of the movie’s 83-minute runtime, which systematically label the most salient narrative and visual elements, from characters and locations to camera cuts and main character speech. This Neurodata Without Borders-formatted dataset contains the spike times and corresponding mean waveforms from all recorded neurons, along with demographic information and electrode localizations. For technical validation, we provide spike-sorting metrics and demonstrate the tuning of individual neurons to movie features. We additionally provide decoding results from a machine learning-based pipeline for predicting movie features from neuronal population activity. To facilitate immediate use, we offer the dataset in a machine learning-ready format alongside a codebase demonstrating both neural feature and movie feature prediction tasks. This dataset offers a valuable foundation for exploring how the human brain processes semantic content, particularly in real-world contexts involving dynamic and naturalistic stimuli.
- Open Access
- 9 May 2026
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (6) , 263
A Guide to Structureless Visual Localization
Visual localization algorithms, i.e., methods that estimate the camera pose of a query image in a known scene, are core components of many applications, including self-driving cars and augmented / mixed reality systems. State-of-the-art visual localization algorithms are structure-based, i.e., they store a 3D model of the scene and use 2D-3D correspondences between the query image and 3D points in the model for camera pose estimation. While such approaches are highly accurate, they are also rather inflexible when it comes to adjusting the underlying 3D model after changes in the scene. Structureless localization approaches represent the scene as a database of images with known poses and thus offer a much more flexible representation that can be easily updated by adding or removing images. Although there is a large amount of literature on structure-based approaches, there is significantly less work on structureless methods. Hence, this paper is dedicated to providing the, to the best of our knowledge, first comprehensive discussion and comparison of structureless methods. Extensive experiments show that approaches that use a higher degree of classical geometric reasoning generally achieve higher pose accuracy. In particular, approaches based on classical absolute or semi-generalized relative pose estimation outperform very recent methods based on pose regression by a wide margin. Compared with state-of-the-art structure-based approaches, the flexibility of structureless methods comes at the cost of (slightly) lower pose accuracy, indicating an interesting direction for future work.
- 9 May 2026
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 134 (6) , 263
Open AccessA Guide to Structureless Visual Localization
Visual localization algorithms, i.e., methods that estimate the camera pose of a query image in a known scene, are core components of many applications, including self-driving cars and augmented / mixed reality systems. State-of-the-art visual localization algorithms are structure-based, i.e., they store a 3D model of the scene and use 2D-3D correspondences between the query image and 3D points in the model for camera pose estimation. While such approaches are highly accurate, they are also rather inflexible when it comes to adjusting the underlying 3D model after changes in the scene. Structureless localization approaches represent the scene as a database of images with known poses and thus offer a much more flexible representation that can be easily updated by adding or removing images. Although there is a large amount of literature on structure-based approaches, there is significantly less work on structureless methods. Hence, this paper is dedicated to providing the, to the best of our knowledge, first comprehensive discussion and comparison of structureless methods. Extensive experiments show that approaches that use a higher degree of classical geometric reasoning generally achieve higher pose accuracy. In particular, approaches based on classical absolute or semi-generalized relative pose estimation outperform very recent methods based on pose regression by a wide margin. Compared with state-of-the-art structure-based approaches, the flexibility of structureless methods comes at the cost of (slightly) lower pose accuracy, indicating an interesting direction for future work.
- Open Access
- 13 August 2025
- Published by eLife Sciences Publications, Ltd in eLife preprints
Decoding movie content from neuronal population activity in the human medial temporal lobe
The human medial temporal lobe (MTL), a region implicated in memory and high-level cognition, contains neurons that respond selectively to stimuli belonging to specific categories, such as individual people, landmarks, or objects. However, these neurons have been largely studied via static, isolated presentations of stimuli. Therefore, it is unclear how neurons in the MTL respond to rich stimuli such as movies, and which dynamical stimulus features can be retrieved from neuronal population spiking activity. We studied single-unit responses from 2286 neurons recorded from the amygdala, hippocampus, entorhinal cortex, and parahippocampal cortex of 29 intracranially implanted patients during the presentation of an 83-minute movie. We found only a few individual neurons that exhibited a classic selective response to semantic features. However, we successfully decoded the presence of characters, settings, and visual transitions from neuronal population activity. The information relevant for decoding varies across regions depending on the feature category, as visual transitions could be decoded from subsets of neurons with selective responses, whereas character and location features relied on distributed representations. Our results demonstrate an approach for reliably decoding movie features in the human MTL, and suggest that the brain uses a population code when representing character and location features.
- 13 August 2025
- Published by eLife Sciences Publications, Ltd in eLife preprints
Open AccessDecoding movie content from neuronal population activity in the human medial temporal lobe
The human medial temporal lobe (MTL), a region implicated in memory and high-level cognition, contains neurons that respond selectively to stimuli belonging to specific categories, such as individual people, landmarks, or objects. However, these neurons have been largely studied via static, isolated presentations of stimuli. Therefore, it is unclear how neurons in the MTL respond to rich stimuli such as movies, and which dynamical stimulus features can be retrieved from neuronal population spiking activity. We studied single-unit responses from 2286 neurons recorded from the amygdala, hippocampus, entorhinal cortex, and parahippocampal cortex of 29 intracranially implanted patients during the presentation of an 83-minute movie. We found only a few individual neurons that exhibited a classic selective response to semantic features. However, we successfully decoded the presence of characters, settings, and visual transitions from neuronal population activity. The information relevant for decoding varies across regions depending on the feature category, as visual transitions could be decoded from subsets of neurons with selective responses, whereas character and location features relied on distributed representations. Our results demonstrate an approach for reliably decoding movie features in the human MTL, and suggest that the brain uses a population code when representing character and location features.
- Open Access
A Guide to Structureless Visual Localization
Visual localization algorithms, i.e., methods that estimate the camera pose of a query image in a known scene, are core components of many applications, including self-driving cars and augmented / mixed reality systems. State-of-the-art visual localization algorithms are structure-based, i.e., they store a 3D model of the scene and use 2D-3D correspondences between the query image and 3D points in the model for camera pose estimation. While such approaches are highly accurate, they are also rather inflexible when it comes to adjusting the underlying 3D model after changes in the scene. Structureless localization approaches represent the scene as a database of images with known poses and thus offer a much more flexible representation that can be easily updated by adding or removing images. Although there is a large amount of literature on structure-based approaches, there is significantly less work on structureless methods. Hence, this paper is dedicated to providing the, to the best of our knowledge, first comprehensive discussion and comparison of structureless methods. Extensive experiments show that approaches that use a higher degree of classical geometric reasoning generally achieve higher pose accuracy. In particular, approaches based on classical absolute or semi-generalized relative pose estimation outperform very recent methods based on pose regression by a wide margin. Compared with state-of-the-art structure-based approaches, the flexibility of structureless methods comes at the cost of (slightly) lower pose accuracy, indicating an interesting direction for future work.
Open AccessA Guide to Structureless Visual Localization
Visual localization algorithms, i.e., methods that estimate the camera pose of a query image in a known scene, are core components of many applications, including self-driving cars and augmented / mixed reality systems. State-of-the-art visual localization algorithms are structure-based, i.e., they store a 3D model of the scene and use 2D-3D correspondences between the query image and 3D points in the model for camera pose estimation. While such approaches are highly accurate, they are also rather inflexible when it comes to adjusting the underlying 3D model after changes in the scene. Structureless localization approaches represent the scene as a database of images with known poses and thus offer a much more flexible representation that can be easily updated by adding or removing images. Although there is a large amount of literature on structure-based approaches, there is significantly less work on structureless methods. Hence, this paper is dedicated to providing the, to the best of our knowledge, first comprehensive discussion and comparison of structureless methods. Extensive experiments show that approaches that use a higher degree of classical geometric reasoning generally achieve higher pose accuracy. In particular, approaches based on classical absolute or semi-generalized relative pose estimation outperform very recent methods based on pose regression by a wide margin. Compared with state-of-the-art structure-based approaches, the flexibility of structureless methods comes at the cost of (slightly) lower pose accuracy, indicating an interesting direction for future work.
- 3 November 2024
- Published by Springer Nature
- p. 108-127
The NeRFect Match: Exploring NeRF Features for Visual Localization
- 3 November 2024
- Published by Springer Nature
- p. 108-127
The NeRFect Match: Exploring NeRF Features for Visual Localization
- Open Access
Lidar Panoptic Segmentation in an Open World
Addressing Lidar Panoptic Segmentation (LPS ) is crucial for safe deployment of autonomous vehicles. LPS aims to recognize and segment lidar points w.r.t. a pre-defined vocabulary of semantic classes, including thing classes of countable objects (e.g., pedestrians and vehicles) and stuff classes of amorphous regions (e.g., vegetation and road). Importantly, LPS requires segmenting individual thing instances (e.g., every single vehicle). Current LPS methods make an unrealistic assumption that the semantic class vocabulary is fixed in the real open world, but in fact, class ontologies usually evolve over time as robots encounter instances of novel classes that are considered to be unknowns w.r.t. the pre-defined class vocabulary. To address this unrealistic assumption, we study LPS in the Open World (LiPSOW): we train models on a dataset with a pre-defined semantic class vocabulary and study their generalization to a larger dataset where novel instances of thing and stuff classes can appear. This experimental setting leads to interesting conclusions. While prior art train class-specific instance segmentation methods and obtain state-of-the-art results on known classes, methods based on class-agnostic bottom-up grouping perform favorably on classes outside of the initial class vocabulary (i.e., unknown classes). Unfortunately, these methods do not perform on-par with fully data-driven methods on known classes. Our work suggests a middle ground: we perform class-agnostic point clustering and over-segment the input cloud in a hierarchical fashion, followed by binary point segment classification, akin to Region Proposal Network [1]. We obtain the final point cloud segmentation by computing a cut in the weighted hierarchical tree of point segments, independently of semantic classification. Remarkably, this unified approach leads to strong performance on both known and unknown classes.
Open AccessLidar Panoptic Segmentation in an Open World
Addressing Lidar Panoptic Segmentation (LPS ) is crucial for safe deployment of autonomous vehicles. LPS aims to recognize and segment lidar points w.r.t. a pre-defined vocabulary of semantic classes, including thing classes of countable objects (e.g., pedestrians and vehicles) and stuff classes of amorphous regions (e.g., vegetation and road). Importantly, LPS requires segmenting individual thing instances (e.g., every single vehicle). Current LPS methods make an unrealistic assumption that the semantic class vocabulary is fixed in the real open world, but in fact, class ontologies usually evolve over time as robots encounter instances of novel classes that are considered to be unknowns w.r.t. the pre-defined class vocabulary. To address this unrealistic assumption, we study LPS in the Open World (LiPSOW): we train models on a dataset with a pre-defined semantic class vocabulary and study their generalization to a larger dataset where novel instances of thing and stuff classes can appear. This experimental setting leads to interesting conclusions. While prior art train class-specific instance segmentation methods and obtain state-of-the-art results on known classes, methods based on class-agnostic bottom-up grouping perform favorably on classes outside of the initial class vocabulary (i.e., unknown classes). Unfortunately, these methods do not perform on-par with fully data-driven methods on known classes. Our work suggests a middle ground: we perform class-agnostic point clustering and over-segment the input cloud in a hierarchical fashion, followed by binary point segment classification, akin to Region Proposal Network [1]. We obtain the final point cloud segmentation by computing a cut in the weighted hierarchical tree of point segments, independently of semantic classification. Remarkably, this unified approach leads to strong performance on both known and unknown classes.
- 19 September 2024
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (3) , 1153-1174
Lidar Panoptic Segmentation in an Open World
- 19 September 2024
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (3) , 1153-1174
Lidar Panoptic Segmentation in an Open World
- Open Access
Decoding movie content from neuronal population activity in the human medial temporal lobe
The human medial temporal lobe (MTL), a region implicated in memory and high-level cognition, contains neurons that respond selectively to stimuli belonging to specific categories, such as individual people, landmarks, or objects. However, these neurons have been largely studied via static, isolated presentations of stimuli. Therefore, it is unclear how neurons in the MTL respond to rich stimuli such as movies, and which dynamical stimulus features can be retrieved from neuronal population spiking activity. We studied single-unit responses from 2286 neurons recorded from the amygdala, hippocampus, entorhinal cortex, and parahippocampal cortex of 29 intracranially implanted patients during the presentation of an 83-minute movie. We found only a few individual neurons that exhibited a classic selective response to semantic features. However, we successfully decoded the presence of characters, settings, and visual transitions from neuronal population activity. The information relevant for decoding varies across regions depending on the feature category, as visual transitions could be decoded from subsets of neurons with selective responses, whereas character and location features relied on distributed representations. Our results demonstrate an approach for reliably decoding movie features in the human MTL, and suggest that the brain uses a population code when representing character and location features.
Open AccessDecoding movie content from neuronal population activity in the human medial temporal lobe
The human medial temporal lobe (MTL), a region implicated in memory and high-level cognition, contains neurons that respond selectively to stimuli belonging to specific categories, such as individual people, landmarks, or objects. However, these neurons have been largely studied via static, isolated presentations of stimuli. Therefore, it is unclear how neurons in the MTL respond to rich stimuli such as movies, and which dynamical stimulus features can be retrieved from neuronal population spiking activity. We studied single-unit responses from 2286 neurons recorded from the amygdala, hippocampus, entorhinal cortex, and parahippocampal cortex of 29 intracranially implanted patients during the presentation of an 83-minute movie. We found only a few individual neurons that exhibited a classic selective response to semantic features. However, we successfully decoded the presence of characters, settings, and visual transitions from neuronal population activity. The information relevant for decoding varies across regions depending on the feature category, as visual transitions could be decoded from subsets of neurons with selective responses, whereas character and location features relied on distributed representations. Our results demonstrate an approach for reliably decoding movie features in the human MTL, and suggest that the brain uses a population code when representing character and location features.
- Open Access
Soft Augmentation for Image Classification
Modern neural networks are over-parameterized and thus rely on strong regularization such as data augmentation and weight decay to reduce overfitting and improve generalization. The dominant form of data augmentation applies invariant transforms, where the learning target of a sample is invariant to the transform applied to that sample. We draw inspiration from human visual classification studies and propose generalizing augmentation with invariant transforms to soft augmentation where the learning target softens non-linearly as a function of the degree of the transform applied to the sample: e.g., more aggressive image crop augmentations produce less confident learning targets. We demonstrate that soft targets allow for more aggressive data augmentation, offer more robust performance boosts, work with other augmentation policies, and interestingly, produce better calibrated models (since they are trained to be less confident on aggressively cropped/occluded examples). Combined with existing aggressive augmentation strategies, soft target 1) doubles the top-1 accuracy boost across Cifar-10, Cifar-100, ImageNet-1K, and ImageNet-V2, 2) improves model occlusion performance by up to $4\times$, and 3) halves the expected calibration error (ECE). Finally, we show that soft augmentation generalizes to self-supervised classification tasks. Code available at https://github.com/youngleox/soft_augmentationOpen AccessSoft Augmentation for Image Classification
Modern neural networks are over-parameterized and thus rely on strong regularization such as data augmentation and weight decay to reduce overfitting and improve generalization. The dominant form of data augmentation applies invariant transforms, where the learning target of a sample is invariant to the transform applied to that sample. We draw inspiration from human visual classification studies and propose generalizing augmentation with invariant transforms to soft augmentation where the learning target softens non-linearly as a function of the degree of the transform applied to the sample: e.g., more aggressive image crop augmentations produce less confident learning targets. We demonstrate that soft targets allow for more aggressive data augmentation, offer more robust performance boosts, work with other augmentation policies, and interestingly, produce better calibrated models (since they are trained to be less confident on aggressively cropped/occluded examples). Combined with existing aggressive augmentation strategies, soft target 1) doubles the top-1 accuracy boost across Cifar-10, Cifar-100, ImageNet-1K, and ImageNet-V2, 2) improves model occlusion performance by up to $4\times$, and 3) halves the expected calibration error (ECE). Finally, we show that soft augmentation generalizes to self-supervised classification tasks. Code available at https://github.com/youngleox/soft_augmentation - Open Access
Staged Contact-Aware Global Human Motion Forecasting
Scene-aware global human motion forecasting is critical for manifold applications, including virtual reality, robotics, and sports. The task combines human trajectory and pose forecasting within the provided scene context, which represents a significant challenge. So far, only Mao et al. NeurIPS'22 have addressed scene-aware global motion, cascading the prediction of future scene contact points and the global motion estimation. They perform the latter as the end-to-end forecasting of future trajectories and poses. However, end-to-end contrasts with the coarse-to-fine nature of the task and it results in lower performance, as we demonstrate here empirically. We propose a STAGed contact-aware global human motion forecasting STAG, a novel three-stage pipeline for predicting global human motion in a 3D environment. We first consider the scene and the respective human interaction as contact points. Secondly, we model the human trajectory forecasting within the scene, predicting the coarse motion of the human body as a whole. The third and last stage matches a plausible fine human joint motion to complement the trajectory considering the estimated contacts. Compared to the state-of-the-art (SoA), STAG achieves a 1.8% and 16.2% overall improvement in pose and trajectory prediction, respectively, on the scene-aware GTA-IM dataset. A comprehensive ablation study confirms the advantages of staged modeling over end-to-end approaches. Furthermore, we establish the significance of a newly proposed temporal counter called the "time-to-go", which tells how long it is before reaching scene contact and endpoints. Notably, STAG showcases its ability to generalize to datasets lacking a scene and achieves a new state-of-the-art performance on CMU-Mocap, without leveraging any social cues. Our code is released at: https://github.com/L-Scofano/STAG
Open AccessStaged Contact-Aware Global Human Motion Forecasting
Scene-aware global human motion forecasting is critical for manifold applications, including virtual reality, robotics, and sports. The task combines human trajectory and pose forecasting within the provided scene context, which represents a significant challenge. So far, only Mao et al. NeurIPS'22 have addressed scene-aware global motion, cascading the prediction of future scene contact points and the global motion estimation. They perform the latter as the end-to-end forecasting of future trajectories and poses. However, end-to-end contrasts with the coarse-to-fine nature of the task and it results in lower performance, as we demonstrate here empirically. We propose a STAGed contact-aware global human motion forecasting STAG, a novel three-stage pipeline for predicting global human motion in a 3D environment. We first consider the scene and the respective human interaction as contact points. Secondly, we model the human trajectory forecasting within the scene, predicting the coarse motion of the human body as a whole. The third and last stage matches a plausible fine human joint motion to complement the trajectory considering the estimated contacts. Compared to the state-of-the-art (SoA), STAG achieves a 1.8% and 16.2% overall improvement in pose and trajectory prediction, respectively, on the scene-aware GTA-IM dataset. A comprehensive ablation study confirms the advantages of staged modeling over end-to-end approaches. Furthermore, we establish the significance of a newly proposed temporal counter called the "time-to-go", which tells how long it is before reaching scene contact and endpoints. Notably, STAG showcases its ability to generalize to datasets lacking a scene and achieves a new state-of-the-art performance on CMU-Mocap, without leveraging any social cues. Our code is released at: https://github.com/L-Scofano/STAG