Sandrine Tornay
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Sandrine Tornay has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Mathew Magimai. -Doss
IDIAP Research Institute, Martigny, Switzerland
3 shared publications - Richard Bowden
University of Surrey
3 shared publications - Alessia Battisti
Department of Computational Linguistics, University of Zurich, Zurich, Switzerland
2 shared publications - Franz Holzknecht
University of Teacher Education in Special Needs (HfH), Zürich, Switzerland
2 shared publications - Katja Tissi
University of Teacher Education in Special Needs (HfH), Zürich, Switzerland
2 shared publications - Neha Tarigopula
Idiap Research Institute, Martigny, Switzerland and École polytechnique fédérale de Lausanne (EPFL), Lausanne, Switzerland
2 shared publications
Latest Publications
- Open Access
- 30 September 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Accessible Computing
- Vol. 19 (3) , 1-47
Web SMILE Demo: A Web Application Providing Automated Feedback on Sign Language Vocabulary Production
In language learning, learners need to develop competence in comprehension and production, both of which are necessary for successful interaction. The use of digital technologies to aid the acquisition of such competencies has proven effective in spoken language learning and is emerging in sign language learning. Most existing sign language learning tools are designed for comprehension acquisition, while the production side involves only self-comparison. However, for learning effective sign language production, good proprioception, spatial reasoning, and observation skills are necessary. There is a need to develop applications that guide learners in various aspects of sign language production. For this reason, we developed an AI-driven, web-based sign language learning application in which sign language production is assessed automatically using neural networks and hidden Markov models. The application guides the learners by providing video feedback and scores at different levels, namely, at (i) the sign level, (ii) the form level (e.g., handshape correctness), and (iii) the spatiotemporal level. This article describes the development process of the application, including its architecture, interface, and AI methods. It also covers the intermediate stages, including evaluation studies of both the UI and the underlying AI model. These studies were conducted with sign language learners and human raters to validate the AI methods underlying the application and feedback designs, and to demonstrate the feasibility of the proposed system. The present study underscores the challenges and key elements to consider when designing such evaluation tools. These elements include the choice of sign language assessment algorithms, the importance of linguistically valid data, the design of effective feedback, the importance of incorporating learners’ perspectives, and utilizing human ratings to validate the tool.
- 30 September 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Accessible Computing
- Vol. 19 (3) , 1-47
Open AccessWeb SMILE Demo: A Web Application Providing Automated Feedback on Sign Language Vocabulary Production
In language learning, learners need to develop competence in comprehension and production, both of which are necessary for successful interaction. The use of digital technologies to aid the acquisition of such competencies has proven effective in spoken language learning and is emerging in sign language learning. Most existing sign language learning tools are designed for comprehension acquisition, while the production side involves only self-comparison. However, for learning effective sign language production, good proprioception, spatial reasoning, and observation skills are necessary. There is a need to develop applications that guide learners in various aspects of sign language production. For this reason, we developed an AI-driven, web-based sign language learning application in which sign language production is assessed automatically using neural networks and hidden Markov models. The application guides the learners by providing video feedback and scores at different levels, namely, at (i) the sign level, (ii) the form level (e.g., handshape correctness), and (iii) the spatiotemporal level. This article describes the development process of the application, including its architecture, interface, and AI methods. It also covers the intermediate stages, including evaluation studies of both the UI and the underlying AI model. These studies were conducted with sign language learners and human raters to validate the AI methods underlying the application and feedback designs, and to demonstrate the feasibility of the proposed system. The present study underscores the challenges and key elements to consider when designing such evaluation tools. These elements include the choice of sign language assessment algorithms, the importance of linguistically valid data, the design of effective feedback, the importance of incorporating learners’ perspectives, and utilizing human ratings to validate the tool.
- Open Access
- 17 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 6, 284-292
Posterior-Based Analysis of Spatio-Temporal Features for Sign Language Assessment
Sign Language conveys information through multiple channels composed of manual (handshape, hand movement) and non-manual (facial expression, mouthing, body posture) components. Sign language assessment involves giving granular feedback to a learner, in terms of correctness of the manual and non-manual components, aiding the learner's progress. Existing methods rely on handcrafted skeleton-based features for hand movement within a KL-HMM framework to identify errors in manual components. However, modern deep learning models offer powerful spatio-temporal representations for videos to represent hand movement and facial expressions. Despite their success in classification tasks, these representations often struggle to attribute errors to specific sources, such as incorrect handshape, improper movement, or incorrect facial expressions. To address this limitation, we leverage and analyze the spatio-temporal representations from Inflated 3D Convolutional Networks (I3D) and integrate them into the KL-HMM framework to assess sign language videos on both manual and non-manual components. By applying masking and cropping techniques, we isolate and evaluate distinct channels of hand movement, and facial expressions using the I3D model and handshape using the CNN-based model. Our approach outperforms traditional methods based on handcrafted features, as validated through experiments on the SMILE-DSGS dataset, and therefore demonstrates that it can enhance the effectiveness of sign language learning tools.
- 17 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 6, 284-292
Open AccessPosterior-Based Analysis of Spatio-Temporal Features for Sign Language Assessment
Sign Language conveys information through multiple channels composed of manual (handshape, hand movement) and non-manual (facial expression, mouthing, body posture) components. Sign language assessment involves giving granular feedback to a learner, in terms of correctness of the manual and non-manual components, aiding the learner's progress. Existing methods rely on handcrafted skeleton-based features for hand movement within a KL-HMM framework to identify errors in manual components. However, modern deep learning models offer powerful spatio-temporal representations for videos to represent hand movement and facial expressions. Despite their success in classification tasks, these representations often struggle to attribute errors to specific sources, such as incorrect handshape, improper movement, or incorrect facial expressions. To address this limitation, we leverage and analyze the spatio-temporal representations from Inflated 3D Convolutional Networks (I3D) and integrate them into the KL-HMM framework to assess sign language videos on both manual and non-manual components. By applying masking and cropping techniques, we isolate and evaluate distinct channels of hand movement, and facial expressions using the I3D model and handshape using the CNN-based model. Our approach outperforms traditional methods based on handcrafted features, as validated through experiments on the SMILE-DSGS dataset, and therefore demonstrates that it can enhance the effectiveness of sign language learning tools.
- Open Access
- 26 May 2024
- Published by Taylor & Francis in Language Assessment Quarterly
- Vol. 21 (3) , 245-265
Automated Sign Language Vocabulary Assessment: Comparing Human and Machine Ratings and Studying Learner Perceptions
- 26 May 2024
- Published by Taylor & Francis in Language Assessment Quarterly
- Vol. 21 (3) , 245-265
Open AccessAutomated Sign Language Vocabulary Assessment: Comparing Human and Machine Ratings and Studying Learner Perceptions
- Open Access
- 26 September 2019
- Published by MDPI AG in Information
- Vol. 10 (10) , 298
Subunits Inference and Lexicon Development Based on Pairwise Comparison of Utterances and Signs
Communication languages convey information through the use of a set of symbols or units. Typically, this unit is word. When developing language technologies, as words in a language do not have the same prior probability, there may not be sufficient training data for each word to model. Furthermore, the training data may not cover all possible words in the language. Due to these data sparsity and word unit coverage issues, language technologies employ modeling of subword units or subunits, which are based on prior linguistic knowledge. For instance, development of speech technologies such as automatic speech recognition system presume that there exists a phonetic dictionary or at least a writing system for the target language. Such knowledge is not available for all languages in the world. In that direction, this article develops a hidden Markov model-based abstract methodology to extract subword units given only pairwise comparison between utterances (or realizations of words in the mode of communication), i.e., whether two utterances correspond to the same word or not. We validate the proposed methodology through investigations on spoken language and sign language. In the case of spoken language, we demonstrate that the proposed methodology can lead up to discovery of phone set and development of phonetic dictionary. In the case of sign language, we demonstrate how hand movement information can be effectively modeled for sign language processing and synthesized back to gain insight about the derived subunits.
- 26 September 2019
- Published by MDPI AG in Information
- Vol. 10 (10) , 298
Open AccessSubunits Inference and Lexicon Development Based on Pairwise Comparison of Utterances and Signs
Communication languages convey information through the use of a set of symbols or units. Typically, this unit is word. When developing language technologies, as words in a language do not have the same prior probability, there may not be sufficient training data for each word to model. Furthermore, the training data may not cover all possible words in the language. Due to these data sparsity and word unit coverage issues, language technologies employ modeling of subword units or subunits, which are based on prior linguistic knowledge. For instance, development of speech technologies such as automatic speech recognition system presume that there exists a phonetic dictionary or at least a writing system for the target language. Such knowledge is not available for all languages in the world. In that direction, this article develops a hidden Markov model-based abstract methodology to extract subword units given only pairwise comparison between utterances (or realizations of words in the mode of communication), i.e., whether two utterances correspond to the same word or not. We validate the proposed methodology through investigations on spoken language and sign language. In the case of spoken language, we demonstrate that the proposed methodology can lead up to discovery of phone set and development of phonetic dictionary. In the case of sign language, we demonstrate how hand movement information can be effectively modeled for sign language processing and synthesized back to gain insight about the derived subunits.
- 1 May 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2817-2821
HMM-based Approaches to Model Multichannel Information in Sign Language Inspired from Articulatory Features-based Speech Processing
Sign language conveys information through multiple channels, such as hand shape, hand movement, and mouthing. Modeling this multichannel information is a highly challenging problem. In this paper, we elucidate the link between spoken language and sign language in terms of production phenomenon and perception phenomenon. Through this link we show that hidden Markov model-based approaches developed to model "articulatory" features for spoken language processing can be exploited to model the multichannel information inherent in sign language for sign language processing.
- 1 May 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2817-2821
HMM-based Approaches to Model Multichannel Information in Sign Language Inspired from Articulatory Features-based Speech Processing
Sign language conveys information through multiple channels, such as hand shape, hand movement, and mouthing. Modeling this multichannel information is a highly challenging problem. In this paper, we elucidate the link between spoken language and sign language in terms of production phenomenon and perception phenomenon. Through this link we show that hidden Markov model-based approaches developed to model "articulatory" features for spoken language processing can be exploited to model the multichannel information inherent in sign language for sign language processing.