DYDi Yang

Di Yang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Di Yang has a total of 21 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Dual the Reasoning, Double the Insight with TambI: A Self-Supervised Framework for Skeleton Action Representation

    by Mahmoud Ali, Snehashis Majhi, Di Yang, Quan Kong, Gianpiero Francesca, Francois Bremond

  • Learning effective video representations for action recognition

    by Di Yang

    Apprendre des représentations vidéo efficaces pour la reconnaissance d'actions: La reconnaissance des actions humaines est un domaine de recherche actif avec d'importantes contributions dans des applications telles que la surveillance à domicile, l'interaction homme-machine et le contrôle de jeux. Cependant, la reconnaissance des activités humaines dans des vidéos du monde réel reste un défi en matière d'apprentissage de représentations vidéo efficaces qui ont un pouvoir expressif élevé pour représenter le mouvement spatio-temporel humain, les actions invariantes par rapport à la vue, les actions complexes composables, etc. Pour relever ce défi, cette thèse apporte trois contributions à l'apprentissage de telles représentations vidéo efficaces. Les représentations apprises peuvent être appliquées et évaluées dans des tâches de classification et de segmentation d'actions humaines du monde réel par apprentissage par transfert. La première contribution vise à améliorer la généralisabilité des modèles de représentation du mouvement du squelette humain. Nous proposons un cadre unifié pour la reconnaissance des actions humaines du monde réel basée sur le squelette. Le cadre comprend un modèle de squelette novateur qui apprend efficacement des caractéristiques spatio-temporelles sur des séquences de squelette humain et généralise également entre les ensembles de données. De plus, le cadre introduit un ensemble de données de pré-entraînement novateur à grande échelle pour améliorer les représentations du squelette et bénéficier de l'apprentissage par transfert dans les tâches ultérieures de reconnaissance d'actions. La deuxième contribution étend le cadre proposé en introduisant deux nouveaux modèles d'apprentissage de la génération d'actions et de représentation conjointe du squelette pour différentes tâches ultérieures. La première méthode est un autoencodeur invariant à la vue pour l'apprentissage de la représentation d'actions du squelette en auto-supervisé, qui peut traiter de grandes variations entre les sujets et les points de vue de la caméra. La deuxième méthode est un cadre auto-supervisé pour l'apprentissage à partir de mouvements composables synthétisés pour la segmentation des actions basées sur le squelette. La troisième contribution porte sur la reconnaissance d'actions vidéo basée sur RGB générale. Plus précisément, une stratégie d'apprentissage par contraste paramétré par le temps est proposée. Elle capture les mouvements liés au temps pour améliorer la performance de la classification d'actions dans des tâches fines et orientées vers l'humain. Les résultats expérimentaux sur des ensembles de données de référence montrent que les approches proposées atteignent des performances de pointe dans les tâches de classification et de segmentation d'actions. Les modèles proposés améliorent la précision et l'interprétabilité de la reconnaissance des activités humaines et fournissent des informations sur la structure sous-jacente et la dynamique des actions humaines dans les vidéos. En fin de compte, cette thèse contribue au domaine de la compréhension des vidéos en proposant des méthodes novatrices pour l'apprentissage de la représentation d'actions basées sur le squelette et pour l'apprentissage de la représentation de vidéos RVB générales. De telles représentations bénéficient à la fois de la classification et de la segmentation d'actions.

  • Open Access

    Are Visual-Language Models Effective in Action Recognition? A Comparative Study

    by Mahmoud Ali, Di Yang, François Brémond

    Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.


  • LIA: Latent Image Animator

    by Yaohui Wang, Yaohui Wang, Yaohui Wang, Yaohui Wang, Di Yang, Di Yang, Di Yang, Di Yang, Francois Bremond, Francois Bremond, et al.

    Previous animation techniques mainly focus on leveraging explicit structure representations ( e.g. , meshes or keypoints) for transferring motion from driving videos to source images. However, such methods are challenged with large appearance variations between source and driving data, as well as require complex additional modules to respectively model appearance and motion. Towards addressing these issues, we introduce the Latent Image Animator (LIA), streamlined to animate high-resolution images. LIA is designed as a simple autoencoder that does not rely on explicit representations. Motion transfer in the pixel space is modeled as linear navigation of motion codes in the latent space. Specifically such navigation is represented as an orthogonal motion dictionary learned in a self-supervised manner based on proposed Linear Motion Decomposition (LMD). Extensive experimental results demonstrate that LIA outperforms state-of-the-art on VoxCeleb, TaichiHD, and TED-talk datasets with respect to video quality and spatio-temporal consistency. In addition LIA is well equipped for zero-shot high-resolution image animation. Code, models, and demo video are available at https://github.com/wyhsirius/LIA .


  • Open Access

    GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields

    by Arnab Dey, Di Yang, Rohith Agaram, Antitza Dantcheva, Andrew I. Comport, Srinath Sridhar, Jean Martinet

    Recent advances in Neural Radiance Fields (NeRF) have demonstrated promising results in 3D scene representations, including 3D human representations. However, these representations often lack crucial information on the underlying human pose and structure, which is crucial for AR/VR applications and games. In this paper, we introduce a novel approach, termed GHNeRF, designed to address these limitations by learning 2D/3D joint locations of human subjects with NeRF representation. GHNeRF uses a pre-trained 2D encoder streamlined to extract essential human features from 2D images, which are then incorporated into the NeRF framework in order to encode human biomechanic features. This allows our network to simultaneously learn biomechanic features, such as joint locations, along with human geometry and texture. To assess the effectiveness of our method, we conduct a comprehensive comparison with state-of-the-art human NeRF techniques and joint estimation algorithms. Our results show that GHNeRF can achieve state-of-the-art results in near real-time.

  • Open Access

    HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields

    by Arnab Dey, Di Yang, Antitza Dantcheva, Jean Martinet

    In recent advancements in novel view synthesis, generalizable Neural Radiance Fields (NeRF) based methods applied to human subjects have shown remarkable results in generating novel views from few images. However, this generalization ability cannot capture the underlying structural features of the skeleton shared across all instances. Building upon this, we introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features using a pre-trained image encoder. While previous human NeRF methods have shown promising results in the generation of photorealistic virtual avatars, such methods lack underlying human structure or biomechanic features such as skeleton or joint information that are crucial for downstream applications including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D pre-trained foundation models toward learning human features in 3D using neural rendering, and then volume rendering towards generating 2D feature maps. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features. The proposed method is fully differentiable, allowing to successfully learn color, geometry, and human skeleton in a simultaneous manner. This paper presents preliminary results of HFNeRF, illustrating its potential in generating realistic virtual avatars with biomechanic features using NeRF.

  • Open Access

    LAC: Latent Action Composition for Skeleton-based Action Segmentation

    by Di Yang, Yaohui Wang, Antitza Dantcheva, Quan Kong, Lorenzo Garattoni, Gianpiero Francesca, Francois Bremond

    Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC), a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthesized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.

  • View-Invariant Skeleton Action Representation Learning via Motion Retargeting

    by Di Yang, Yaohui Wang, Antitza Dantcheva, Lorenzo Garattoni, Gianpiero Francesca, François Brémond


  • Open Access

    LAC - Latent Action Composition for Skeleton-based Action Segmentation

    by Di Yang, Yaohui Wang, Antitza Dantcheva, Quan Kong, Lorenzo Garattoni, Gianpiero Francesca, François Brémond

    Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC) 1 , a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthe-sized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.