Jean-Claude Martin

Automatic analysis of job interviews has gained in interest amongst academic and industrial research. The particular case of asynchronous video interviews allows to collect vast corpora of videos where candidates answer standardized questions in monologue videos, enabling the use of deep learning algorithms. On the other hand, state-of-the-art approaches still face some obstacles, among which the fusion of information from multiple modalities and the interpretability of the predictions. We study the task of predicting candidates performance in asynchronous video interviews using three modalities (verbal content, prosody and facial expressions) independently or simultaneously, using data from real interviews which take place in real conditions. We propose a sequential and multimodal deep neural network model, called Multimodal HireNet. We compare this model to state-of-the-art approaches and show a clear improvement of the performance. Moreover, the architecture we propose is based on attention mechanism, which provides interpretability about which questions, moments and modalities contribute the most to the output of the network. While other deep learning systems use attention mechanisms to offer a visualization of moments with attention values, the proposed methodology enables an in-depth interpretation of the predictions by an overall analysis of the features of social signals contained in these moments.

We conducted an experimental study to investigate how the participants combine touch and facial expression to evaluate emotional valence. In this study, visual and haptic stimuli were presented separately and then presented together. The visual stimuli comprised pictures of facial expressions with different emotional levels, and the touch stimuli consisted of air jet tactile stimulation performed on the arms of the participants. The participants were asked to evaluate the communicated emotional valence on a continuous scale. Information Integration Theory was used to model the algebraic rule (addition, multiplication, averaging, and so on) that underlies the multimodal perception of emotional valence. Analyses showed that the participants usually integrated both visual and touch information to evaluate the emotional valence. The main integration rule was averaging, with nonsystematic predominance of each modality over the other modality.

In recent years technological devices dedicated to learning and research on learning have been widely developed. This article aims to examine the use of new technologies in research on academic learning and their contributions in terms of control of the experimental device, of development of knowledge on a learner model, and of development of tools such as e-learning platforms. Intellilangue’s project aims to support the learning of English among French junior high school students through a fun platform. This project serves here as a framework for understanding how the mechanisms of evaluation and emulation influence behaviors of learner, thus allowing the comparison of theoretical models in a context of applied research.