Di Yang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Di Yang has a total of 21 co-authors. Here are the top co-authors by number of shared publications:
- Antitza Dantcheva
Inria Center at Université Côte d'Azur
8 shared publications - Francois Bremond
INRIA Sophia Antipolis, Sophia Antipolis, France
7 shared publications - Gianpiero Francesca
Toyota Motor Europe
5 shared publications - Yaohui Wang
Shanghai Artificial Intelligence Laboratory, Shanghai, China
5 shared publications - Lorenzo Garattoni
Toyota Motor Europe, Brussels, Belgium
4 shared publications - Mahmoud Ali
Inria Center at Université Côte d’Azur
3 shared publications
Latest Publications
- 4 August 2026
- Published by Springer Nature
- p. 223-239
Dual the Reasoning, Double the Insight with TambI: A Self-Supervised Framework for Skeleton Action Representation
- 4 August 2026
- Published by Springer Nature
- p. 223-239
Dual the Reasoning, Double the Insight with TambI: A Self-Supervised Framework for Skeleton Action Representation
- 8 April 2026
- Published by Agence Bibliographique de l'Enseignement Supérieur
Learning effective video representations for action recognition
Apprendre des représentations vidéo efficaces pour la reconnaissance d'actions: La reconnaissance des actions humaines est un domaine de recherche actif avec d'importantes contributions dans des applications telles que la surveillance à domicile, l'interaction homme-machine et le contrôle de jeux. Cependant, la reconnaissance des activités humaines dans des vidéos du monde réel reste un défi en matière d'apprentissage de représentations vidéo efficaces qui ont un pouvoir expressif élevé pour représenter le mouvement spatio-temporel humain, les actions invariantes par rapport à la vue, les actions complexes composables, etc. Pour relever ce défi, cette thèse apporte trois contributions à l'apprentissage de telles représentations vidéo efficaces. Les représentations apprises peuvent être appliquées et évaluées dans des tâches de classification et de segmentation d'actions humaines du monde réel par apprentissage par transfert. La première contribution vise à améliorer la généralisabilité des modèles de représentation du mouvement du squelette humain. Nous proposons un cadre unifié pour la reconnaissance des actions humaines du monde réel basée sur le squelette. Le cadre comprend un modèle de squelette novateur qui apprend efficacement des caractéristiques spatio-temporelles sur des séquences de squelette humain et généralise également entre les ensembles de données. De plus, le cadre introduit un ensemble de données de pré-entraînement novateur à grande échelle pour améliorer les représentations du squelette et bénéficier de l'apprentissage par transfert dans les tâches ultérieures de reconnaissance d'actions. La deuxième contribution étend le cadre proposé en introduisant deux nouveaux modèles d'apprentissage de la génération d'actions et de représentation conjointe du squelette pour différentes tâches ultérieures. La première méthode est un autoencodeur invariant à la vue pour l'apprentissage de la représentation d'actions du squelette en auto-supervisé, qui peut traiter de grandes variations entre les sujets et les points de vue de la caméra. La deuxième méthode est un cadre auto-supervisé pour l'apprentissage à partir de mouvements composables synthétisés pour la segmentation des actions basées sur le squelette. La troisième contribution porte sur la reconnaissance d'actions vidéo basée sur RGB générale. Plus précisément, une stratégie d'apprentissage par contraste paramétré par le temps est proposée. Elle capture les mouvements liés au temps pour améliorer la performance de la classification d'actions dans des tâches fines et orientées vers l'humain. Les résultats expérimentaux sur des ensembles de données de référence montrent que les approches proposées atteignent des performances de pointe dans les tâches de classification et de segmentation d'actions. Les modèles proposés améliorent la précision et l'interprétabilité de la reconnaissance des activités humaines et fournissent des informations sur la structure sous-jacente et la dynamique des actions humaines dans les vidéos. En fin de compte, cette thèse contribue au domaine de la compréhension des vidéos en proposant des méthodes novatrices pour l'apprentissage de la représentation d'actions basées sur le squelette et pour l'apprentissage de la représentation de vidéos RVB générales. De telles représentations bénéficient à la fois de la classification et de la segmentation d'actions.
- 8 April 2026
- Published by Agence Bibliographique de l'Enseignement Supérieur
Learning effective video representations for action recognition
Apprendre des représentations vidéo efficaces pour la reconnaissance d'actions: La reconnaissance des actions humaines est un domaine de recherche actif avec d'importantes contributions dans des applications telles que la surveillance à domicile, l'interaction homme-machine et le contrôle de jeux. Cependant, la reconnaissance des activités humaines dans des vidéos du monde réel reste un défi en matière d'apprentissage de représentations vidéo efficaces qui ont un pouvoir expressif élevé pour représenter le mouvement spatio-temporel humain, les actions invariantes par rapport à la vue, les actions complexes composables, etc. Pour relever ce défi, cette thèse apporte trois contributions à l'apprentissage de telles représentations vidéo efficaces. Les représentations apprises peuvent être appliquées et évaluées dans des tâches de classification et de segmentation d'actions humaines du monde réel par apprentissage par transfert. La première contribution vise à améliorer la généralisabilité des modèles de représentation du mouvement du squelette humain. Nous proposons un cadre unifié pour la reconnaissance des actions humaines du monde réel basée sur le squelette. Le cadre comprend un modèle de squelette novateur qui apprend efficacement des caractéristiques spatio-temporelles sur des séquences de squelette humain et généralise également entre les ensembles de données. De plus, le cadre introduit un ensemble de données de pré-entraînement novateur à grande échelle pour améliorer les représentations du squelette et bénéficier de l'apprentissage par transfert dans les tâches ultérieures de reconnaissance d'actions. La deuxième contribution étend le cadre proposé en introduisant deux nouveaux modèles d'apprentissage de la génération d'actions et de représentation conjointe du squelette pour différentes tâches ultérieures. La première méthode est un autoencodeur invariant à la vue pour l'apprentissage de la représentation d'actions du squelette en auto-supervisé, qui peut traiter de grandes variations entre les sujets et les points de vue de la caméra. La deuxième méthode est un cadre auto-supervisé pour l'apprentissage à partir de mouvements composables synthétisés pour la segmentation des actions basées sur le squelette. La troisième contribution porte sur la reconnaissance d'actions vidéo basée sur RGB générale. Plus précisément, une stratégie d'apprentissage par contraste paramétré par le temps est proposée. Elle capture les mouvements liés au temps pour améliorer la performance de la classification d'actions dans des tâches fines et orientées vers l'humain. Les résultats expérimentaux sur des ensembles de données de référence montrent que les approches proposées atteignent des performances de pointe dans les tâches de classification et de segmentation d'actions. Les modèles proposés améliorent la précision et l'interprétabilité de la reconnaissance des activités humaines et fournissent des informations sur la structure sous-jacente et la dynamique des actions humaines dans les vidéos. En fin de compte, cette thèse contribue au domaine de la compréhension des vidéos en proposant des méthodes novatrices pour l'apprentissage de la représentation d'actions basées sur le squelette et pour l'apprentissage de la représentation de vidéos RVB générales. De telles représentations bénéficient à la fois de la classification et de la segmentation d'actions.
- 12 May 2025
- Published by Springer Nature
- p. 46-59
Are Visual-Language Models Effective in Action Recognition? A Comparative Study
- 12 May 2025
- Published by Springer Nature
- p. 46-59
Are Visual-Language Models Effective in Action Recognition? A Comparative Study
- Open Access
Are Visual-Language Models Effective in Action Recognition? A Comparative Study
Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.
Open AccessAre Visual-Language Models Effective in Action Recognition? A Comparative Study
Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.
- 23 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (12) , 10829-10844
LIA: Latent Image Animator
Previous animation techniques mainly focus on leveraging explicit structure representations ( e.g. , meshes or keypoints) for transferring motion from driving videos to source images. However, such methods are challenged with large appearance variations between source and driving data, as well as require complex additional modules to respectively model appearance and motion. Towards addressing these issues, we introduce the Latent Image Animator (LIA), streamlined to animate high-resolution images. LIA is designed as a simple autoencoder that does not rely on explicit representations. Motion transfer in the pixel space is modeled as linear navigation of motion codes in the latent space. Specifically such navigation is represented as an orthogonal motion dictionary learned in a self-supervised manner based on proposed Linear Motion Decomposition (LMD). Extensive experimental results demonstrate that LIA outperforms state-of-the-art on VoxCeleb, TaichiHD, and TED-talk datasets with respect to video quality and spatio-temporal consistency. In addition LIA is well equipped for zero-shot high-resolution image animation. Code, models, and demo video are available at https://github.com/wyhsirius/LIA .
- 23 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (12) , 10829-10844
LIA: Latent Image Animator
Previous animation techniques mainly focus on leveraging explicit structure representations ( e.g. , meshes or keypoints) for transferring motion from driving videos to source images. However, such methods are challenged with large appearance variations between source and driving data, as well as require complex additional modules to respectively model appearance and motion. Towards addressing these issues, we introduce the Latent Image Animator (LIA), streamlined to animate high-resolution images. LIA is designed as a simple autoencoder that does not rely on explicit representations. Motion transfer in the pixel space is modeled as linear navigation of motion codes in the latent space. Specifically such navigation is represented as an orthogonal motion dictionary learned in a self-supervised manner based on proposed Linear Motion Decomposition (LMD). Extensive experimental results demonstrate that LIA outperforms state-of-the-art on VoxCeleb, TaichiHD, and TED-talk datasets with respect to video quality and spatio-temporal consistency. In addition LIA is well equipped for zero-shot high-resolution image animation. Code, models, and demo video are available at https://github.com/wyhsirius/LIA .
- Open Access
GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields
Recent advances in Neural Radiance Fields (NeRF) have demonstrated promising results in 3D scene representations, including 3D human representations. However, these representations often lack crucial information on the underlying human pose and structure, which is crucial for AR/VR applications and games. In this paper, we introduce a novel approach, termed GHNeRF, designed to address these limitations by learning 2D/3D joint locations of human subjects with NeRF representation. GHNeRF uses a pre-trained 2D encoder streamlined to extract essential human features from 2D images, which are then incorporated into the NeRF framework in order to encode human biomechanic features. This allows our network to simultaneously learn biomechanic features, such as joint locations, along with human geometry and texture. To assess the effectiveness of our method, we conduct a comprehensive comparison with state-of-the-art human NeRF techniques and joint estimation algorithms. Our results show that GHNeRF can achieve state-of-the-art results in near real-time.
Open AccessGHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields
Recent advances in Neural Radiance Fields (NeRF) have demonstrated promising results in 3D scene representations, including 3D human representations. However, these representations often lack crucial information on the underlying human pose and structure, which is crucial for AR/VR applications and games. In this paper, we introduce a novel approach, termed GHNeRF, designed to address these limitations by learning 2D/3D joint locations of human subjects with NeRF representation. GHNeRF uses a pre-trained 2D encoder streamlined to extract essential human features from 2D images, which are then incorporated into the NeRF framework in order to encode human biomechanic features. This allows our network to simultaneously learn biomechanic features, such as joint locations, along with human geometry and texture. To assess the effectiveness of our method, we conduct a comprehensive comparison with state-of-the-art human NeRF techniques and joint estimation algorithms. Our results show that GHNeRF can achieve state-of-the-art results in near real-time.
- Open Access
HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields
In recent advancements in novel view synthesis, generalizable Neural Radiance Fields (NeRF) based methods applied to human subjects have shown remarkable results in generating novel views from few images. However, this generalization ability cannot capture the underlying structural features of the skeleton shared across all instances. Building upon this, we introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features using a pre-trained image encoder. While previous human NeRF methods have shown promising results in the generation of photorealistic virtual avatars, such methods lack underlying human structure or biomechanic features such as skeleton or joint information that are crucial for downstream applications including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D pre-trained foundation models toward learning human features in 3D using neural rendering, and then volume rendering towards generating 2D feature maps. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features. The proposed method is fully differentiable, allowing to successfully learn color, geometry, and human skeleton in a simultaneous manner. This paper presents preliminary results of HFNeRF, illustrating its potential in generating realistic virtual avatars with biomechanic features using NeRF.
Open AccessHFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields
In recent advancements in novel view synthesis, generalizable Neural Radiance Fields (NeRF) based methods applied to human subjects have shown remarkable results in generating novel views from few images. However, this generalization ability cannot capture the underlying structural features of the skeleton shared across all instances. Building upon this, we introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features using a pre-trained image encoder. While previous human NeRF methods have shown promising results in the generation of photorealistic virtual avatars, such methods lack underlying human structure or biomechanic features such as skeleton or joint information that are crucial for downstream applications including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D pre-trained foundation models toward learning human features in 3D using neural rendering, and then volume rendering towards generating 2D feature maps. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features. The proposed method is fully differentiable, allowing to successfully learn color, geometry, and human skeleton in a simultaneous manner. This paper presents preliminary results of HFNeRF, illustrating its potential in generating realistic virtual avatars with biomechanic features using NeRF.
- Open Access
LAC: Latent Action Composition for Skeleton-based Action Segmentation
Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC), a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthesized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.
Open AccessLAC: Latent Action Composition for Skeleton-based Action Segmentation
Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC), a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthesized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.
- 16 January 2024
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 132 (7) , 2351-2366
View-Invariant Skeleton Action Representation Learning via Motion Retargeting
- 16 January 2024
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 132 (7) , 2351-2366
View-Invariant Skeleton Action Representation Learning via Motion Retargeting
- Open Access
- 1 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13633-13644
LAC - Latent Action Composition for Skeleton-based Action Segmentation
Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC) 1 , a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthe-sized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.
- 1 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13633-13644
Open AccessLAC - Latent Action Composition for Skeleton-based Action Segmentation
Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC) 1 , a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthe-sized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.