Thomas Thebaud

La protection des données personnelles est un devoir éthique et légal en Europe. Les systèmes d’authentification biométriques sont naturellement concernés : ils utilisent des vecteurs discriminants les utilisateurs extraits à partir de leurs données biométriques. D’un point de vue entre l’académique et l’industriel, cette thèse explore les vulnérabilités de ces vecteurs - embeddings - qui pourraient être utilisées pour voler des données personnelles et frauder un système d’authentification, appliquées à deux biométries comportementales : la parole et les chiffres manuscrits tracés à la main. Les attaques visant à reconstruire ou analyser des embeddings incluent généralement un accès à l’encodeur qui les a produit. Nous avons proposé d’utiliser un alignement statistique non supervisé pour nous permettre, sans accès à l’encodeur, d’analyser un ensemble d’embeddings de chiffres manuscrits volés et de retrouver les chiffres qui leurs étaient associés. En affinant cet alignement et en utilisant un décodeur adapté, nous avons pu reconstruire les tracés de chiffres assez précisément pour frauder le système d’authentification qui les avait produits. La reconstruction de parole est plus complexe, car elle contiens une information linguistique très faiblement conservée dans les embeddings, nous avons donc utilisé un système de conversion de voix pour reconstruire des extraits de paroles assez précis pour frauder un système de vérification du locuteur. Nous avons aussi proposé une attaque sur un système de pseudo anonymisation, en utilisant des alignements supervisés et non supervisés sur les embeddings anonymisés produits par celui-ci, pour explorer ses limites.

Motor dysfunction is a common sign of neurodegenerative diseases (NDs) such as Parkinson's disease (PD) and Alzheimer's disease (AD), but may be difficult to detect, especially in the early stages. In this work, we examine the behavior of a wide array of interpretable features extracted from the handwriting signals of 113 subjects performing multiple tasks on a digital tablet, as part of the Neurological Signals dataset. The aim is to measure their effectiveness in characterizing NDs, including AD and PD. To this end, task-agnostic and task-specific features are extracted from 14 distinct tasks. Subsequently, through statistical analysis and a series of classification experiments, we investigate which features provide greater discriminative power between NDs and healthy controls and amongst different NDs. Preliminary results indicate that the tasks at hand can all be effectively leveraged to distinguish between the considered set of NDs, specifically by measuring the stability, the speed of writing, the time spent not writing, and the pressure variations between groups from our handcrafted interpretable features, which shows a statistically significant difference between groups, across multiple tasks. Using various binary classification algorithms on the computed features, we obtain up to 87% accuracy for the discrimination between AD and healthy controls (CTL), and up to 69% for the discrimination between PD and CTL.

In the realm of audiovisual emotion recognition, a significant challenge lies in developing neural network architectures capable of effectively harnessing and integrating multimodal information. This study introduces an advanced methodology for the Empathic Virtual Agent Challenge (EVAC), utilizing state-of-the-art speech, language, and image models. Specifically, we leverage cutting-edge pre-trained models, including multilingual variants fine-tuned in French for each modality, and integrate them using late fusion techniques. Through extensive experimentation and validation, we demonstrate the efficacy of our approach in achieving competitive results on the challenge dataset. Our findings highlight that multimodal approaches outperform unimodal methods across Core Affect Presence and Intensity and Appraisal Dimensions tasks, underscoring the effectiveness of integrating diverse modalities. This underscores the importance of leveraging multiple sources of information to capture nuanced emotional states more accurately and robustly in real-world applications.

Despite many recent developments in speaker diarization, it remains a challenge and an active area of research to make diarization robust and effective in real-life scenarios. Well-established clustering-based methods are showing good performance and qualities. However, such systems are built of several independent, separately optimized modules, which may cause non-optimum performance. End-to-end neural speaker diarization (EEND) systems are considered the next stepping stone in pursuing high-performance diarization. Nevertheless, this approach also suffers limitations, such as dealing with long recordings and scenarios with a large (more than four) or unknown number of speakers in the recording. The appearance of EEND with encoder-decoder-based attractors (EEND-EDA) enabled us to deal with recordings that contain a flexible number of speakers thanks to an LSTM-based EDA module. A competitive alternative over the referenced EEND-EDA baseline is the EEND with non-autoregressive attractor (EEND-NAA) estimation, proposed recently by the authors of this article. NAA back-end incorporates k-means clustering as part of the attractor estimation and an attractor refinement module based on a Transformer decoder. However, in our previous work on EEND-NAA, we assumed a known number of speakers, and the experimental evaluation was limited to 2-speaker recordings only. In this article, we describe in detail our recent EEND-NAA approach and propose further improvements to the EEND-NAA architecture, introducing three novel variants of the NAA back-end, which can handle recordings containing speech of a variable and unknown number of speakers. Conducted experiments include simulated mixtures generated using the Switchboard and NIST SRE datasets and real-life recordings from the CALLHOME and DIHARD II datasets. In experimental evaluation, the proposed systems achieve up to 51% relative improvement for the simulated scenario and up to 15% for real recordings over the baseline EEND-EDA.

Background: Motor changes are early signs of neurodegenerative diseases such as Alzheimer’s disease (AD), but they are often difficult to detect, especially in the early stages. In the past, the examination of handwriting has been used to investigate motoric changes in different neurodegenerative diseases. In this study, we explore a wide array of explainable characteristics of the handwriting of patients obtained with signal processing, and study their potential for assessment of Mild Cognitive Impairment (MCI) and AD.Method: We collected the written tasks from 37 participants, 7 with AD, 17 with MCI, and 13 for the control group (CTL) on a digital tablet. We analyzed 14 different tasks, including holding the pen over a point, copying images, free writing/drawing, and memory copying. We extracted multiple movement characteristics, including the length of each stroke, the ratio of the time the pen spent on‐air vs in‐tablet and the standard deviation of the distance between consecutive points. The drawings of each subject were manually supervised, then processed to extract multiple characteristics, and compared to identify the ones providing the most significant differences between the three studied groups using Welch’s t‐test, with false discovery rate (FDR) correction.Result: Preliminary results showed that the ratio between the time with the pen in contact with the tablet and without (on‐air vs in‐tablet) for writing tasks, significantly differentiate between AD and CTL patients (p‐value<0.01), as well as the position standard deviation when asked to maintain the pen at a fixed position (p‐value<0.05).The comparison between MCI and CTL populations also showed that the length of the strokes for drawings and copying tasks can deliver clues to distinguish both groups (p‐value<0.05).Conclusion: Automated analysis of a large variety of handwriting segments employing machine learning techniques can provide an assessment of patients with AD and MCI that may be useful in their early diagnosis and assessment of progression.

Our novel technique utilizes a Joint Energy-based Model (JEM) that integrates both discriminative and generative approaches to increase resistance against dirty-label backdoor attacks. Our approach is especially effective when the trigger is short or hardly perceivable. We simulate the attack on the Speech Commands Dataset consisting of 1s audio clips. During training, we use JEM to model a view of the input implemented by a randomly selected 610ms window. During inference, we combine all (40) possible views utilizing a generative part of JEM. The resulting system has slightly decreased accuracy but significantly increased resistance shown in multiple scenarios. Interestingly, replacing JEM with a standard discriminative model (Disc) provides increased resistance with a lesser effect compared to JEM but maintains accuracy. We introduce an extension motivated by semi-supervised training that further improves JEM but not Disc. JEM can also benefit from Gaussian noise during evaluation.