JSJosé Santos-Victor

José Santos-Victor

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


José Santos-Victor has a total of 308 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Extreme driving segmentation for drift detection and characterisation

    by Ines Rito Lima, Yukie Nagai, José Santos-Victor

  • Open Access

    Anticipating Vehicle Drift in the Real-World: A Data-Driven Intelligence Layer for Future AVs

    by Ines Rito Lima, Yukie Nagai, José Santos-Victor

    Extreme driving scenarios, such as drifting, pose significant risks to driving safety. This article proposes a novel intelligence layer for vehicle drift detection, characterisation and anticipation, fostering the development of Advanced Driver Assistance Systems (ADAS) with increased safety for future Autonomous Vehicles (AVs). The intelligence layer leverages real-world data from expert drivers while driving and drifting, and encompasses distinct Machine Learning (ML) models to: i) segment driving states; ii) identify the key features for drift detection and characterisation; iii) define vehicle stability limits; and iv) anticipate vehicle drift through deep learning and time-series forecasting. The proposed solution provides the probability of entering drifting 200 ms into the future, with 94.4% accuracy across distinct driving and drifting states, based on vehicle velocity, sideslip angle and driver’s steering input. The models’ robustness was evaluated across distinct race tracks. The findings provide actionable insights for developing enhancedwarning systems sensitive to increased-risk scenarios and triggering appropriate response mechanisms in AVs. Future lines of research are outlined to integrate expert driver response strategies to support the effective mitigation and resolution of drifting events.

  • Open Access

    Task-oriented grasping for dexterous robots using postural synergies and reinforcement learning

    by Dimitrios Dimou, José Santos-Victor, Plinio Moreno

    In this paper, we address the problem of task-oriented grasping for humanoid robots, emphasizing the need to align with human social norms and task-specific objectives. Existing methods, employ a variety of open-loop and closed-loop approaches but lack an end-to-end solution that can grasp several objects while taking into account the downstream task's constraints. Our proposed approach employs reinforcement learning to enhance task-oriented grasping, prioritizing the post-grasp intention of the agent. We extract human grasp preferences from the ContactPose dataset, and train a hand synergy model based on the Variational Autoencoder (VAE) to imitate the participant's grasping actions. Based on this data, we train an agent able to grasp multiple objects while taking into account distinct post-grasp intentions that are task-specific. By combining data-driven insights from human grasping behavior with learning by exploration provided by reinforcement learning, we can develop humanoid robots capable of context-aware manipulation actions, facilitating collaboration in human-centered environments.

  • Open Access

    The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation

    by João Damião Almeida, Egidio Falotico, Cecilia Laschi, José Santos-Victor

    In-hand manipulation tasks, particularly in human-inspired robotic systems, must rely on distributed tactile sensing to achieve precise control across a wide variety of tasks. However, the optimal configuration of this network of sensors is a complex problem, and while the fingertips are a common choice for placing sensors, the contribution of tactile information from other regions of the hand is often overlooked. This work investigates the impact of tactile feedback from various regions of the fingers and palm in performing in-hand object reorientation tasks. We analyze how sensory feedback from different parts of the hand influences the robustness of deep reinforcement learning control policies and investigate the relationship between object characteristics and optimal sensor placement. We identify which tactile sensing configurations contribute to improving the efficiency and accuracy of manipulation. Our results provide valuable insights for the design and use of anthropomorphic end-effectors with enhanced manipulation capabilities.

  • Driving in the Extreme: Sensing Drifting and the Expert Driver Response

    by Ines Rito Lima, José Santos-Victor

    Understanding the professional response to extreme driving events fosters the creation of novel safety procedures for Autonomous Vehicles (AVs), capable of avoiding and handling critical scenarios. This work proposes a high-frequency sensing system and a data processing methodology for capturing the vehicle dynamics and the associated expert human response in extreme driving conditions, namely drifting. The system resorts to multiple sensors placed across the vehicle to extract data such as the vehicle’s position, acceleration, heading, and the driver’s steering and pedal inputs. The processing pipeline of each sensing component is presented, to extract the above-mentioned features, and combine time-series to detect the occurrence of drifting events, and the patterns that precede and succeed them. The system has been tested in various real-life racing environments and the results demonstrate temporal and spatial coherence with the track challenges and with the intensity of the driving events and responses. The analysis unveils responsive driving behaviours based on the track layout with human response patterns upon critical scenarios. Future analysis of acquired and processed data will target using Machine Learning (ML) to model the driver’s expert behaviour and contribute to enhancing road safety mechanisms.


  • Open Access

    Attention Analysis in Robotic-Assistive Therapy for Children With Autism

    by Bárbara Silva, Laura Santos, Catarina Barata, Alice Geminiani, Gabriele Fassina, Ana Rita Gonzalez, Sara Ferreira, Bernardo Barahona-Corrêa, Ivana Olivieri, Alessandra Pedrocchi, et al.

    Children with Autism Spectrum Disorder (ASD) show severe attention deficits, hindering their capacity to acquire new skills. The automatic assessment of their attention response would provide the therapists with an important biomarker to better quantify their behaviour and monitor their progress during therapy. This work aims to develop a quantitativemodel, to evaluate the attention response of children with ASD, during robotic-assistive therapeutic sessions. Previous attempts to quantify the attention response of autistic subjects during human-robot interaction tasks were limited to restrained child movements. Instead, we developed an accurate quantitative system to assess the attention of ASD children in unconstrained scenarios. Our approach combines gaze extraction (Gaze360 model) with the definition of angular Areas-of-Interest, to characterise periods of attention towards elements of interest in the therapy environment during the session. The methodology was tested with 12 ASD children, achieving a mean test accuracy of 79.5 %. Finally, the proposed attention index was consistent with the therapists’ evaluation of patients, allowing a meaningful interpretation of the automatic evaluation. This encourages the future clinical use of the proposed system.


  • Open Access

    Robotic Learning the Sequence of Packing Irregular Objects from Human Demonstrations

    by André Santos, Nuno Ferreira Duarte, Atabak Dehban, José Santos-Victor

    We tackle the challenge of robotic bin packing with irregular objects, such as groceries. Given the diverse physical attributes of these objects and the complex constraints governing their placement and manipulation, employing preprogrammed strategies becomes unfeasible. Our approach is to learn directly from expert demonstrations in order to extract implicit task knowledge and strategies to ensure safe object positioning, efficient use of space, and the generation of human-like behaviors that enhance human-robot trust. We rely on human demonstrations to learn a Markov chain for predicting the object packing sequence for a given set of items and then compare it with human performance. Our experimental results show that the model outperforms human performance by generating sequence predictions that humans classify as human-like more frequently than human-generated sequences. The human demonstrations were collected using our proposed VR platform, BoxED, which is a box packaging environment for simulating real-world objects and scenarios for fast and streamlined data collection with the purpose of teaching robots. We collected data from 43 participants packing a total of 263 boxes with supermarket-like objects, yielding 4644 object manipulations. Our VR platform can be easily adapted to new scenarios and objects, and is publicly available, alongside our dataset, at https://github.com/andrejfsantos4/BoxED.

  • Differential effects of intrinsic properties of natural scenes and interference mechanisms on recognition processes in long-term visual memory

    by Anastasiia Mikhailova, Sophie Lightfoot, José Santos-Victor, Moreno I. Coco

    Humans display remarkable long-term visual memory (LTVM) processes. Even though images may be intrinsically memorable, the fidelity of their visual representations, and consequently the likelihood of successfully retrieving them, hinges on their similarity when concurrently held in LTVM. In this debate, it is still unclear whether intrinsic features of images (perceptual and semantic) may be mediated by mechanisms of interference generated at encoding, or during retrieval, and how these factors impinge on recognition processes. In the current study, participants (32) studied a stream of 120 natural scenes from 8 semantic categories, which varied in frequencies (4, 8, 16 or 32 exemplars per category) to generate different levels of category interference, in preparation for a recognition test. Then they were asked to indicate which of two images, presented side by side (i.e. two-alternative forced-choice), they remembered. The two images belonged to the same semantic category but varied in their perceptual similarity (similar or dissimilar). Participants also expressed their confidence (sure/not sure) about their recognition response, enabling us to tap into their metacognitive efficacy (meta-d’). Additionally, we extracted the activation of perceptual and semantic features in images (i.e. their informational richness) through deep neural network modelling and examined their impact on recognition processes. Corroborating previous literature, we found that category interference and perceptual similarity negatively impact recognition processes, as well as response times and metacognitive efficacy. Moreover, images semantically rich were less likely remembered, an effect that trumped a positive memorability boost coming from perceptual information. Critically, we did not observe any significant interaction between intrinsic features of images and interference generated either at encoding or during retrieval. All in all, our study calls for a more integrative understanding of the representational dynamics during encoding and recognition enabling us to form, maintain and access visual information.


  • Open Access

    Robotic hand synergies for in-hand regrasping driven by object information

    by Dimitrios Dimou, José Santos-Victor, Plinio Moreno

    We develop a conditional generative model to represent dexterous grasp postures of a robotic hand and use it to generate in-hand regrasp trajectories. Our model learns to encode the robotic grasp postures into a low-dimensional space, called Synergy Space, while taking into account additional information about the object such as its size and its shape category. We then generate regrasp trajectories through linear interpolation in this low-dimensional space. The result is that the hand configuration moves from one grasp type to another while keeping the object stable in the hand. We show that our model achieves higher success rate on in-hand regrasping compared to previous methods used for synergy extraction, by taking advantage of the grasp size conditional variable.


  • The Gaze Dialogue Model: Nonverbal Communication in HHI and HRI

    by Mirko Raković, Nuno Ferreira Duarte, Jorge Marques, Aude Billard, José Santos-Victor

    When humans interact with each other, eye gaze movements have to support motor control as well as communication. On the one hand, we need to fixate the task goal to retrieve visual information required for safe and precise action-execution. On the other hand, gaze movements fulfil the purpose of communication, both for reading the intention of our interaction partners, as well as to signal our action intentions to others. We study this Gaze Dialogue between two participants working on a collaborative task involving two types of actions: 1) individual action and 2) action-in-interaction . We recorded the eye-gaze data of both participants during the interaction sessions in order to build a computational model, the Gaze Dialogue , encoding the interplay of the eye movements during the dyadic interaction. The model also captures the correlation between the different gaze fixation points and the nature of the action. This knowledge is used to infer the type of action performed by an individual. We validated the model against the recorded eye-gaze behavior of one subject, taking the eye-gaze behavior of the other subject as the input. Finally, we used the model to design a humanoid robot controller that provides interpersonal gaze coordination in human–robot interaction scenarios. During the interaction, the robot is able to: 1) adequately infer the human action from gaze cues; 2) adjust its gaze fixation according to the human eye-gaze behavior; and 3) signal nonverbal cues that correlate with the robot’s own action intentions.