VCViviane Clay

Viviane Clay

*This is an algorithmically generated author record based on publicly available information.

Viviane Clay has a total of 19 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Thousand-Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inference

    by Niels Leadholm, Viviane Clay, Scott Knudstrup, Hojae Lee, Jeff Hawkins

    Current AI systems achieve impressive performance on many tasks, yet they lack core attributes of biological intelligence, including rapid, continual learning, representations grounded in sensorimotor interactions, and structured knowledge that enables efficient generalization. Neuroscience theory suggests that mammals evolved flexible intelligence through the replication of a semi-independent, sensorimotor module, a functional unit known as a cortical column. To address the disparity between biological and artificial intelligence, Thousand Brains systems were proposed as a means of mirroring the architecture of cortical columns and their interactions. In our current work, we evaluate the unique properties of Monty, the first implementation of a Thousand Brains system. We focus on 3D object perception and, in particular, the combined task of object recognition and pose estimation. Utilizing the YCB data set of household objects, we first assess Monty’s use of sensorimotor learning to build structured representations, finding that these enable robust generalization. These representations include an emphasis on classifying objects by their global shape, as well as a natural ability to detect object symmetries. We then explore Monty’s use of model-free and model-based policies to enable rapid inference by supporting principled movements. We find that such policies complement Monty’s modular architecture, a design that can accommodate communication between modules to further accelerate inference speed via a novel voting algorithm. Finally, we examine Monty’s use of associative, Hebbian-like binding to enable rapid, continual, and computationally efficient learning, properties that compare favorably to current deep learning architectures. While Monty is still in a nascent stage of development, these findings support Thousand Brains systems as a powerful and promising new approach to AI and reinforce the importance of sensorimotor learning for developing intelligent systems.


  • Open Access

    Hierarchy or Heterarchy? A Theory of Long-Range Connections for the Sensorimotor Brain

    by Jeff Hawkins, Niels Leadholm, Viviane Clay

    In the traditional understanding of the neocortex, sensory information flows up a hierarchy of regions, with each level processing increasingly complex features. Information also flows down the hierarchy via a different set of connections. Although the hierarchical model has significant support, many anatomical connections do not conform to the standard hierarchical interpretation. In addition, hierarchically arranged regions sometimes respond in parallel, not sequentially as would occur in a hierarchy. This and other evidence suggests that two regions can act in parallel and hierarchically at the same time. Given this flexibility, the word "heterarchy" might be a more suitable term to describe neocortical organization. This paper proposes a new interpretation of how sensory and motor information is processed in the neocortex. The key to our proposal is what we call the "Thousand Brains Theory", which posits that every cortical column is a sensorimotor learning system. Columns learn by integrating sensory input over multiple movements of a sensor. In this view, even primary and secondary regions, such as V1 and V2, can learn and recognize complete 3D objects. This suggests that the hierarchical connections between regions are used to learn the compositional structure of parent objects composed of smaller child objects. We explain the theory by examining the different types of long-range connections between cortical regions and between the neocortex and thalamus. We describe these connections, and then suggest the specific roles they play in the context of a heterarchy of sensorimotor regions. We also suggest that the thalamus plays an essential role in transforming the pose between objects and sensors. The novel perspective we argue for here has broad implications for both neuroscience and artificial intelligence.


  • Open Access

    Thousand-Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inference

    by Niels Leadholm, Viviane Clay, Scott Knudstrup, Hojae Lee, Jeff Hawkins

    Current AI systems achieve impressive performance on many tasks, yet they lack core attributes of biological intelligence, including rapid, continual learning, representations grounded in sensorimotor interactions, and structured knowledge that enables efficient generalization. Neuroscience theory suggests that mammals evolved flexible intelligence through the replication of a semi-independent, sensorimotor module, a functional unit known as a cortical column. To address the disparity between biological and artificial intelligence, thousand-brains systems were proposed as a means of mirroring the architecture of cortical columns and their interactions. In the current work, we evaluate the unique properties of Monty, the first implementation of a thousand-brains system. We focus on 3D object perception, and in particular, the combined task of object recognition and pose estimation. Utilizing the YCB dataset of household objects, we first assess Monty's use of sensorimotor learning to build structured representations, finding that these enable robust generalization. These representations include an emphasis on classifying objects by their global shape, as well as a natural ability to detect object symmetries. We then explore Monty's use of model-free and model-based policies to enable rapid inference by supporting principled movements. We find that such policies complement Monty's modular architecture, a design that can accommodate communication between modules to further accelerate inference speed via a novel `voting' algorithm. Finally, we examine Monty's use of associative, Hebbian-like binding to enable rapid, continual, and computationally efficient learning, properties that compare favorably to current deep learning architectures. While Monty is still in a nascent stage of development, these findings support thousand-brains systems as a powerful and promising new approach to AI.

  • Open Access

    The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence

    by Viviane Clay, Niels Leadholm, Jeff Hawkins

    Artificial intelligence has advanced rapidly in the last decade, driven primarily by progress in the scale of deep-learning systems. Despite these advances, the creation of intelligent systems that can operate effectively in diverse, real-world environments remains a significant challenge. In this white paper, we outline the Thousand Brains Project, an ongoing research effort to develop an alternative, complementary form of AI, derived from the operating principles of the neocortex. We present an early version of a thousand-brains system, a sensorimotor agent that is uniquely suited to quickly learn a wide range of tasks and eventually implement any capabilities the human neocortex has. Core to its design is the use of a repeating computational unit, the learning module, modeled on the cortical columns found in mammalian brains. Each learning module operates as a semi-independent unit that can model entire objects, represents information through spatially structured reference frames, and both estimates and is able to effect movement in the world. Learning is a quick, associative process, similar to Hebbian learning in the brain, and leverages inductive biases around the spatial structure of the world to enable rapid and continual learning. Multiple learning modules can interact with one another both hierarchically and non-hierarchically via a "cortical messaging protocol" (CMP), creating more abstract representations and supporting multimodal integration. We outline the key principles motivating the design of thousand-brains systems and provide details about the implementation of Monty, our first instantiation of such a system. Code can be found at https://github.com/thousandbrainsproject/tbp.monty, along with more detailed documentation at https://thousandbrainsproject.readme.io/.

  • Open Access

    Does Bodily Action Shape Spatial Representation? Evidence from Virtual Reality, Sensory Augmentation and Map Learning

    by Nicolas Kuske, Viviane Clay

    Spatial relations can be defined with respect to the body (egocentric) or among environmental objects only (allocentric). Egocentric relations are necessarily transformed through bodily action. To what extent allocentric cognitive representations are shaped by the body remains unclear. In our study, participants navigate a virtual-reality (VR) city over multiple days in one of three embodiment conditions. In two VR conditions, the participants sit on a swivel chair actively changing navigation direction through bodily rotation. In one of these groups the VR participants wear a sensory augmentation belt which indicates the cardinal direction of north through vibration. The third group of participants navigates a two-dimensional map of the city. After each exploration session, participants complete tasks asking for allocentric spatial relations. We find that the performance in the spatial tasks interacts with the duration of exploration time and the embodiment condition. These findings indicate allocentric spatial representations to be structured by bodily action.


  • Open Access

    Development of Few-Shot Learning Capabilities in Artificial Neural Networks When Learning Through Self-Supervised Interaction

    by Viviane Clay, Gordon Pipa, Kai-Uwe Kühnberger, Peter König

    Most artificial neural networks used for object recognition are trained in a fully supervised setup. This is not only resource consuming as it requires large data sets of labeled examples but also quite different from how humans learn. We use a setup in which an artificial agent first learns in a simulated world through self-supervised, curiosity-driven exploration. Following this initial learning phase, the learned representations can be used to quickly associate semantic concepts such as different types of doors using one or more labeled examples. To do this, we use a method we call fast concept mapping which uses correlated firing patterns of neurons to define and detect semantic concepts. This association works instantaneously with very few labeled examples, similar to what we observe in humans in a phenomenon called fast mapping . Strikingly, we can already identify objects with as little as one labeled example which highlights the quality of the encoding learned self-supervised through interaction with the world. It therefore presents a feasible strategy for learning concepts without much supervision and shows that through pure interaction meaningful representations of an environment can be learned that work better for few-short learning than non-interactive methods.


  • Open Access

    Learning sparse and meaningful representations through embodiment

    by Viviane Clay, Peter König, Kai-Uwe Kühnberger, Gordon Pipa


  • Open Access

    A quantitative analysis of the taxonomy of artistic styles

    by Viviane Clay, Johannes Schrumpf, Yannick Tessenow, Helmut Leder, Ulrich Ansorge, Peter König

    Classifying artists and their work as distinct art styles has been an important task of scholars in the field of art history. Due to its subjectivity, scholars often contradict one another. Our project investigated differences in aesthetic qualities of seven art styles through quantitative means. This was achieved with state-of-the-art deep-learning paradigms to generate new images resembling the style of an artist or entire era. We conducted psychological experiments to measure the behavior of subjects when viewing these new art images. Two different experiments were used: In an eye-tracking study, subjects viewed art-style-specific generated images. Eye movements were recorded and then compared between art styles. In a visual singleton search study, subjects had to locate a style-outlier image among three images of an alternative style. Reaction time and accuracy were measured and analyzed. These experiments show that there are measurable differences in behavior when viewing images of varying art styles. From these differences, we constructed hierarchical clusterings relating art styles based on the different behaviors of subjects viewing the samples. Our study reveals a novel perspective on the classification of artworks into stylistic eras and motivates future research in the domain of empirical aesthetics through quantitative means.


  • Open Access

    Embodied Spatial Knowledge Acquisition in Immersive Virtual Reality: Comparison to Map Exploration

    by Sabine U. König, Ashima Keshava, Viviane Clay, Kirsten Rittershofer, Nicolas Kuske, Peter König

    Investigating spatial learning in virtual environments allows studying different sources of information under controlled conditions. We built a virtual environment in the style of a European village and investigated spatial knowledge acquisition by experience in the virtual environment and by the use of an interactive map. We tested knowledge of cardinal directions, building-to-building orientation, and judgment of direction between buildings. We find that judgment of directions was more accurate after virtual reality exploration than after map exploration, and the opposite results were observed for knowledge of cardinal directions and relative orientation between buildings. Further, the alignment effect was confined to the map exploration condition. Taken together, our results suggest that the source of spatial exploration differentially influenced spatial knowledge acquisition.


  • Open Access

    Learning of Spatial Properties of a Large-scale Virtual City with an Interactive Map

    by Sabine U. König, Viviane Kakerbeck, Debora Nolte, Laura Duesberg, Nicolas Kuske, Peter König

    On the basis of embodied/-enacted theories of the mind, investigations of spatial cognition related to real world environments have become current research interests. How this perspective relates to acquiring spatial knowledge not by active exploration, but through map learning, however, remains unresolved. Therefore, we designed a large virtual city comprised of over 200 houses, suitable for active exploration or spatial learning based on a map. Here, we report our results after single and repeated 30-minute training sessions using an interactive city map. We tested subjects’ knowledge of the orientation of houses towards cardinal north and of two houses relative to each other and the locations of two houses towards each other in a pointing task. Our results revealed that a single training session was sufficient to repeatedly view the majority of houses covering a large area of the city. However, repeated training sessions were necessary to improve the performance level and reveal significant differences between tasks. In contrast to a previous study in a real world city, performance in the two orientation tasks was better than in the pointing task. The lack of distance and the lack of angular difference effects onto task performance suggest the use of a global reference frame. Performance was positively correlated with a self-report on spatial abilities (FRS) in the absolute orientation and pointing task but not the relative orientation task. Overall, our results suggest that training with an interactive city map enhances abstract knowledge, not directly available from an embodied perspective.