Jianyang Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Jianyang Shi has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Haijun Zhang
Harbin Institute of Technology
5 shared publications - Jianghong Ma
Shenzhen Key Laboratory of Internet Information Collaboration, Shenzhen, China, Harbin Institute of Technology (Shenzhen), Shenzhen, China, and City University of Hong Kong, Shenzhen Research Institute, Shenzhen, China
3 shared publications - Jie Hou
Harbin Institute of Technology, Shenzhen
2 shared publications - Xiangyu Mu
Harbin Institute of Technology
2 shared publications - Yimin Yang
Department of Electrical and Computer Engineering, Western University, London, ON N6A 3K7, Canada
2 shared publications - Dongliang Zhou
Tianjin University, Tianjin, China
1 shared publication
Latest Publications
- Open Access
- 20 February 2024
- Published by Springer Nature in Artificial Intelligence Review
- Vol. 57 (3) , 1-41
Fashion intelligence in the Metaverse: promise and future prospects
With the development of artificial intelligence (AI) and the constraints on offline activities imposed due to the sudden outbreak of the COVID epidemic, the Metaverse has recently attracted significant research attention from both academia and industrial practitioners. Fashion, as an expression of a consumer’s aesthetics and personality, has enormous economic potential in both the real world and the Metaverse. In this research, we provide a comprehensive survey of two of the most important components of fashion in the Metaverse: virtual digital humans, and tasks related to fashion items. We survey state-of-the-art articles from 2007 to the present and provide a new taxonomy of extant research topics based on these articles. We also highlight the applications of these topics in the Metaverse from the perspectives of designers and consumers. Finally, we describe possible scenes involving fashion in the Metaverse. The current challenges and open issues related to the fashion industry in the Metaverse are also discussed in order to provide guidance for fashion practitioners, and to shed some light on the future development of fashion AI in the Metaverse.
- 20 February 2024
- Published by Springer Nature in Artificial Intelligence Review
- Vol. 57 (3) , 1-41
Open AccessFashion intelligence in the Metaverse: promise and future prospects
With the development of artificial intelligence (AI) and the constraints on offline activities imposed due to the sudden outbreak of the COVID epidemic, the Metaverse has recently attracted significant research attention from both academia and industrial practitioners. Fashion, as an expression of a consumer’s aesthetics and personality, has enormous economic potential in both the real world and the Metaverse. In this research, we provide a comprehensive survey of two of the most important components of fashion in the Metaverse: virtual digital humans, and tasks related to fashion items. We survey state-of-the-art articles from 2007 to the present and provide a new taxonomy of extant research topics based on these articles. We also highlight the applications of these topics in the Metaverse from the perspectives of designers and consumers. Finally, we describe possible scenes involving fashion in the Metaverse. The current challenges and open issues related to the fashion industry in the Metaverse are also discussed in order to provide guidance for fashion practitioners, and to shed some light on the future development of fashion AI in the Metaverse.
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 7152-7163
Toward Intelligent Interactive Design: A Generation Framework Based on Cross-domain Fashion Elements
Traditional fashion design typically requires the expertise of designers, which limits the involvement of ordinary users during the design process. While it would be desirable for users to participate in the preliminary design phase, their lack of basic design knowledge may render them too inexperienced to produce satisfactory designs. To improve the design efficiency for common users, we present a novel interactive fashion design framework based on generative adversarial network (GAN). This framework can assist users in designing fashion items by drawing only rough scribbles and providing simple fashion styles. Specifically, we propose a new cross-domain feature fusion encoder network that maps design image features from different domains into a series of style vectors which are then fed into a generator. We demonstrate that the learned style vectors can decouple the representations of cross-domain design elements and control the design results through scribbles and style images. Furthermore, we propose a method for rewriting our model with scribbles and style images, to allow designers to train our model more easily. To examine the effectiveness of our proposed model, we constructed a large-scale dataset containing 90,000 pairs of fashion item images. Experimental results show that our proposed method outperforms state-of-the-art methods and can effectively control cross-domain image features, suggesting the potential of our model for providing users with an intelligence-driven interactive design tool.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 7152-7163
Open AccessToward Intelligent Interactive Design: A Generation Framework Based on Cross-domain Fashion Elements
Traditional fashion design typically requires the expertise of designers, which limits the involvement of ordinary users during the design process. While it would be desirable for users to participate in the preliminary design phase, their lack of basic design knowledge may render them too inexperienced to produce satisfactory designs. To improve the design efficiency for common users, we present a novel interactive fashion design framework based on generative adversarial network (GAN). This framework can assist users in designing fashion items by drawing only rough scribbles and providing simple fashion styles. Specifically, we propose a new cross-domain feature fusion encoder network that maps design image features from different domains into a series of style vectors which are then fed into a generator. We demonstrate that the learned style vectors can decouple the representations of cross-domain design elements and control the design results through scribbles and style images. Furthermore, we propose a method for rewriting our model with scribbles and style images, to allow designers to train our model more easily. To examine the effectiveness of our proposed model, we constructed a large-scale dataset containing 90,000 pairs of fashion item images. Experimental results show that our proposed method outperforms state-of-the-art methods and can effectively control cross-domain image features, suggesting the potential of our model for providing users with an intelligence-driven interactive design tool.
- Open Access
- 25 February 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 19 (3) , 1-23
Toward Intelligent Fashion Design: A Texture and Shape Disentangled Generative Adversarial Network
Texture and shape in fashion, constituting essential elements of garments, characterize the body and surface of the fabric and outline the silhouette of clothing, respectively. The selection of texture and shape plays a critical role in the design process, as they largely determine the success of a new design for fashion items. In this research, we propose a texture and shape disentangled generative adversarial network (TSD-GAN) to perform “intelligent” design with the transformation of texture and shape in fashion items. Our TSD-GAN aims to learn how to disentangle the features of texture and shape of different fashion items in an unsupervised manner. Specifically, a fashion attribute encoder is developed to decompose the input fashion items into independent representations of texture and shape. Then, to learn the coarse or fine styles hidden in the features of texture and shape, a texture mapping network and a shape mapping network are proposed to disentangle the features into different hierarchical representations. The different hierarchical representations of texture and shape are then fed into a multi-factor-based generator to generate mixed-style fashion items. In addition, a multi-discriminator framework is developed to distinguish the authenticity and texture similarity between the generated images and the real images. Experimental results on different fashion categories demonstrate that our proposed TSD-GAN may be useful for assisting designers to accomplish the design process by transforming the texture and shape of fashion items.
- 25 February 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 19 (3) , 1-23
Open AccessToward Intelligent Fashion Design: A Texture and Shape Disentangled Generative Adversarial Network
Texture and shape in fashion, constituting essential elements of garments, characterize the body and surface of the fabric and outline the silhouette of clothing, respectively. The selection of texture and shape plays a critical role in the design process, as they largely determine the success of a new design for fashion items. In this research, we propose a texture and shape disentangled generative adversarial network (TSD-GAN) to perform “intelligent” design with the transformation of texture and shape in fashion items. Our TSD-GAN aims to learn how to disentangle the features of texture and shape of different fashion items in an unsupervised manner. Specifically, a fashion attribute encoder is developed to decompose the input fashion items into independent representations of texture and shape. Then, to learn the coarse or fine styles hidden in the features of texture and shape, a texture mapping network and a shape mapping network are proposed to disentangle the features into different hierarchical representations. The different hierarchical representations of texture and shape are then fed into a multi-factor-based generator to generate mixed-style fashion items. In addition, a multi-discriminator framework is developed to distinguish the authenticity and texture similarity between the generated images and the real images. Experimental results on different fashion categories demonstrate that our proposed TSD-GAN may be useful for assisting designers to accomplish the design process by transforming the texture and shape of fashion items.
- Open Access
- 21 February 2023
- Published by Springer Nature in Research Square
Fashion Intelligence in the Metaverse: Promise and Future Prospects
With the development of artificial intelligence (AI) and the constraints on offline activities imposed due to the sudden outbreak of the COVID epidemic, the Metaverse has recently attracted significant research attention from both academia and industrial practitioners. Fashion, as an expression of a consumer’s aesthetics and personality, has enormous economic potential in both the real world and the Metaverse. In this research, we provide a comprehensive survey of two of the most important components of fashion in the Metaverse: virtual digital humans, and tasks related to fashion items. We survey state-of-the-art articles from 2007 to the present and provide a new taxonomy of extant research topics based on these articles. We also highlight the applications of these topics in the Metaverse from the perspectives of designers and consumers. Finally, we describe possible scenes involving fashion in the Metaverse. The current challenges and open issues related to the fashion industry in the Metaverse are also discussed in order to provide guidance for fashion practitioners, and to shed some light on the future development of fashion AI in the Metaverse.
- 21 February 2023
- Published by Springer Nature in Research Square
Open AccessFashion Intelligence in the Metaverse: Promise and Future Prospects
With the development of artificial intelligence (AI) and the constraints on offline activities imposed due to the sudden outbreak of the COVID epidemic, the Metaverse has recently attracted significant research attention from both academia and industrial practitioners. Fashion, as an expression of a consumer’s aesthetics and personality, has enormous economic potential in both the real world and the Metaverse. In this research, we provide a comprehensive survey of two of the most important components of fashion in the Metaverse: virtual digital humans, and tasks related to fashion items. We survey state-of-the-art articles from 2007 to the present and provide a new taxonomy of extant research topics based on these articles. We also highlight the applications of these topics in the Metaverse from the perspectives of designers and consumers. Finally, we describe possible scenes involving fashion in the Metaverse. The current challenges and open issues related to the fashion industry in the Metaverse are also discussed in order to provide guidance for fashion practitioners, and to shed some light on the future development of fashion AI in the Metaverse.
- 18 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 7545-7558
DETA: A Point-Based Tracker With Deformable Transformer and Task-Aligned Learning
Current point-based trackers are usually implemented by the following two branches: a classification branch for predicting the target candidate locations and a regression branch for regressing the tracking box, which may lead to a spatial misalignment between the two tasks. Meanwhile, they ignore a meaningful exploration on how to define positive and negative samples during training and explicit border information for accurate box prediction. In this research, we investigate the key issues of point-based trackers and unlock their key limitations. First, we design a novel task-aligned component and a new loss function, named task-aligned loss, to learn the alignment of the classification and regression tasks. Second, we introduce a border alignment (BorderAlign) component in both the classification and regression branches to effectively exploit the border features of a tracking target. Third, we develop an adaptive training sample assignment (ATSA) to adaptively divide the positive and negative samples based on the statistical characteristics of the tracking object. Finally, a deformable transformer is developed to enhance the representations of search features and explore rich temporal contexts among video frames. Extensive experimental results demonstrate that the proposed tracker achieves state-of-the-art performance on six tracking benchmark datasets.
- 18 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 7545-7558
DETA: A Point-Based Tracker With Deformable Transformer and Task-Aligned Learning
Current point-based trackers are usually implemented by the following two branches: a classification branch for predicting the target candidate locations and a regression branch for regressing the tracking box, which may lead to a spatial misalignment between the two tasks. Meanwhile, they ignore a meaningful exploration on how to define positive and negative samples during training and explicit border information for accurate box prediction. In this research, we investigate the key issues of point-based trackers and unlock their key limitations. First, we design a novel task-aligned component and a new loss function, named task-aligned loss, to learn the alignment of the classification and regression tasks. Second, we introduce a border alignment (BorderAlign) component in both the classification and regression branches to effectively exploit the border features of a tracking target. Third, we develop an adaptive training sample assignment (ATSA) to adaptively divide the positive and negative samples based on the statistical characteristics of the tracking object. Finally, a deformable transformer is developed to enhance the representations of search features and explore rich temporal contexts among video frames. Extensive experimental results demonstrate that the proposed tracker achieves state-of-the-art performance on six tracking benchmark datasets.