Jinhui Yu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jinhui Yu has a total of 43 co-authors. Here are the top co-authors by number of shared publications:
- Jing Liao
City University of Hong Kong
4 shared publications - Kang Zhang
Hong Kong University of Science and Technology, Guangzhou, Guangzhou, China
3 shared publications - Hongwei Lin
School of Mathematical Sciences Zhejiang University 310058 Hangzhou China
2 shared publications - Jiajing Zhang
Department of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou, China
2 shared publications - Junsong Zhang
School of Informatics, Xiamen University, Xiamen, China
2 shared publications - Mohan Zhang
Tencent, Shenzhen, China
2 shared publications
Latest Publications
- Open Access
- 31 July 2022
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Applied Perception
- Vol. 19 (3) , 1-21
Creating Word Paintings Jointly Considering Semantics, Attention, and Aesthetics
In this article, we present a content-aware method for generating a word painting. Word painting is a composite artwork made from the assemblage of words extracted from a given text, which carries similar semantics and visual features to a given source image. However, word painting, usually created by skilled artists, involves tedious manual processes, especially when generating streamlines and laying out text. Hence, we provide an easy method to create word paintings for users. How to design textural layout that simultaneously conveys the input image and enables easy access to the semantic theme is the key challenge to generating a visually pleasing word painting. To address this issue, given an image and its content-related text, we first decompose the input image into several regions and approximate each region with a smooth vector field. At the same time, by analyzing the input text, we extract some weighted keywords as the graphic elements. Then, to measure the likelihood of positions in the input image that attract the observers’ attention, we generate a saliency map with our trained visual attention model. Finally, jointly considering visual attention and aesthetic rules, we propose an energy-based optimization framework to arrange extracted keywords into the decomposed regions and synthesize a word painting. Experimental results and user studies show that this method is able to generate a fashionable and appealing word painting.
- 31 July 2022
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Applied Perception
- Vol. 19 (3) , 1-21
Open AccessCreating Word Paintings Jointly Considering Semantics, Attention, and Aesthetics
In this article, we present a content-aware method for generating a word painting. Word painting is a composite artwork made from the assemblage of words extracted from a given text, which carries similar semantics and visual features to a given source image. However, word painting, usually created by skilled artists, involves tedious manual processes, especially when generating streamlines and laying out text. Hence, we provide an easy method to create word paintings for users. How to design textural layout that simultaneously conveys the input image and enables easy access to the semantic theme is the key challenge to generating a visually pleasing word painting. To address this issue, given an image and its content-related text, we first decompose the input image into several regions and approximate each region with a smooth vector field. At the same time, by analyzing the input text, we extract some weighted keywords as the graphic elements. Then, to measure the likelihood of positions in the input image that attract the observers’ attention, we generate a saliency map with our trained visual attention model. Finally, jointly considering visual attention and aesthetic rules, we propose an energy-based optimization framework to arrange extracted keywords into the decomposed regions and synthesize a word painting. Experimental results and user studies show that this method is able to generate a fashionable and appealing word painting.
The Computer-Based Generation of Fonts in the Style of Kandinsky
This article presents a general framework for programmed automatic generation of artistic fonts. By parameterizing various font attributes, such as color and aspect ratio, the authors are able to generate artistically styled fonts in almost unlimited variations to suit any type of design requirement. The authors demonstrate their experiments on generating fonts in an abstract style similar to Kandinsky's, built on a collection of the artist's styled patterns. The approach generates fonts composed of vector strokes and is thus highly scalable, limited only by the computer hardware.
The Computer-Based Generation of Fonts in the Style of Kandinsky
This article presents a general framework for programmed automatic generation of artistic fonts. By parameterizing various font attributes, such as color and aspect ratio, the authors are able to generate artistically styled fonts in almost unlimited variations to suit any type of design requirement. The authors demonstrate their experiments on generating fonts in an abstract style similar to Kandinsky's, built on a collection of the artist's styled patterns. The approach generates fonts composed of vector strokes and is thus highly scalable, limited only by the computer hardware.
- Top Cited PapersOpen Access
- 24 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 77164-77187
A Comprehensive Survey on Computational Aesthetic Evaluation of Visual Art Images: Metrics and Challenges
Computational image aesthetic evaluation is a computable human aesthetic perception and judgment realized by machines, which has a significant impact on a variety of applications such as image advanced search and promotional exhibition of painting arts. Various approaches have been proposed in copious literature trying to solve this challenging problem. However, there have been few attempts in reviewing works from different types of visual arts, due to their significant differences in visual features and aesthetic principles. In this survey, we present a comprehensive listing of the reviewed works on aesthetic assessment of photographs and paintings, mainly highlighting the contributions and innovations of the existing approaches. We firstly introduce aesthetic assessment benchmark datasets in different categories. Then, conventional aesthetic evaluation approaches based on handcrafted features are reviewed. Besides, we systematically evaluate recent deep learning techniques that are useful for developing robust models for aesthetic prediction tasks in scoring, distribution, attribute, and description. Moreover, the possibility of aesthetic-aware color enhancement, recomposition of photo images, and automatic generation of aesthetic-guided art paintings through computational approaches are summarized. Finally, challenges and potential future directions for this field are discussed. We hope that our survey could serve as a comprehensive reference source for future research on computational aesthetics in visual media.
- 24 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 77164-77187
Top Cited PapersOpen AccessA Comprehensive Survey on Computational Aesthetic Evaluation of Visual Art Images: Metrics and Challenges
Computational image aesthetic evaluation is a computable human aesthetic perception and judgment realized by machines, which has a significant impact on a variety of applications such as image advanced search and promotional exhibition of painting arts. Various approaches have been proposed in copious literature trying to solve this challenging problem. However, there have been few attempts in reviewing works from different types of visual arts, due to their significant differences in visual features and aesthetic principles. In this survey, we present a comprehensive listing of the reviewed works on aesthetic assessment of photographs and paintings, mainly highlighting the contributions and innovations of the existing approaches. We firstly introduce aesthetic assessment benchmark datasets in different categories. Then, conventional aesthetic evaluation approaches based on handcrafted features are reviewed. Besides, we systematically evaluate recent deep learning techniques that are useful for developing robust models for aesthetic prediction tasks in scoring, distribution, attribute, and description. Moreover, the possibility of aesthetic-aware color enhancement, recomposition of photo images, and automatic generation of aesthetic-guided art paintings through computational approaches are summarized. Finally, challenges and potential future directions for this field are discussed. We hope that our survey could serve as a comprehensive reference source for future research on computational aesthetics in visual media.
- 6 January 2021
- Published by Springer Nature in The Visual Computer
- Vol. 38 (2) , 419-432
Visual complexity of shapes: a hierarchical perceptual learning model
- 6 January 2021
- Published by Springer Nature in The Visual Computer
- Vol. 38 (2) , 419-432
Visual complexity of shapes: a hierarchical perceptual learning model
- Top Cited PapersOpen Access
- 14 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 225857-225871
Inkthetics: A Comprehensive Computational Model for Aesthetic Evaluation of Chinese Ink Paintings
Assessing the aesthetic appeal of artworks has become an active research direction recently. However, previous works mainly focus on photographs and oil paintings, there have been few attempts in predicting aesthetics of Chinese ink paintings, due to their significant differences in visual features, semantic features, and aesthetic principles. Aiming at this problem, we propose a comprehensive framework, named Inkthetics, to quantify aesthetics of Chinese ink paintings based on deep learning. Firstly, an aesthetic assessment dataset is built for Chinese ink painting images. Secondly, a deep multi-view parallel convolutional neural network (DMVCNN) is designed by extracting global attribute images and multi-patches as inputs to jointly learn aesthetic features. Finally, we build a comprehensive aesthetic evaluation model by fusing the deeply-learned features with handcrafted features that rely on art expert knowledge. Experimental results show that our proposed deep network significantly outperforms existing methods on the dataset, and our proposed model can predict human aesthetic judgment with Pearson highly significant correlation of 0.843, which indicates an improvement up to 5.7% than the DMVCNN model when the handcrafted features are fused with activation from DMVCNN. Our work not only provides a deep-learning-based reference framework for computational aesthetic evaluation of Chinese paintings, but also explores to what extent can handcrafted features aid learning-based features in predicting human aesthetic perceptions.
- 14 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 225857-225871
Top Cited PapersOpen AccessInkthetics: A Comprehensive Computational Model for Aesthetic Evaluation of Chinese Ink Paintings
Assessing the aesthetic appeal of artworks has become an active research direction recently. However, previous works mainly focus on photographs and oil paintings, there have been few attempts in predicting aesthetics of Chinese ink paintings, due to their significant differences in visual features, semantic features, and aesthetic principles. Aiming at this problem, we propose a comprehensive framework, named Inkthetics, to quantify aesthetics of Chinese ink paintings based on deep learning. Firstly, an aesthetic assessment dataset is built for Chinese ink painting images. Secondly, a deep multi-view parallel convolutional neural network (DMVCNN) is designed by extracting global attribute images and multi-patches as inputs to jointly learn aesthetic features. Finally, we build a comprehensive aesthetic evaluation model by fusing the deeply-learned features with handcrafted features that rely on art expert knowledge. Experimental results show that our proposed deep network significantly outperforms existing methods on the dataset, and our proposed model can predict human aesthetic judgment with Pearson highly significant correlation of 0.843, which indicates an improvement up to 5.7% than the DMVCNN model when the handcrafted features are fused with activation from DMVCNN. Our work not only provides a deep-learning-based reference framework for computational aesthetic evaluation of Chinese paintings, but also explores to what extent can handcrafted features aid learning-based features in predicting human aesthetic perceptions.
- 1 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 28 (8) , 2926-2937
Deep Exemplar-Based Color Transfer for 3D Model
Recoloring 3D models is a challenging task that often requires professional knowledge and tedious manual efforts. In this article, we present the first deep-learning framework for exemplar-based 3D model recolor, which can automatically transfer the colors from a reference image to the 3D model texture. Our framework consists of two modules to solve two major challenges in the 3D color transfer. First, we propose a new feed-forward Color Transfer Network to achieve high-quality semantic-level color transfer by finding dense semantic correspondences between images. Second, considering 3D model constraints such as UV mapping, we design a novel 3D Texture Optimization Module which can generate a seamless and coherent texture by combining color transferred results rendered in multiple views. Experiments show that our method performs robustly and generalizes well to various kinds of models.
- 1 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 28 (8) , 2926-2937
Deep Exemplar-Based Color Transfer for 3D Model
Recoloring 3D models is a challenging task that often requires professional knowledge and tedious manual efforts. In this article, we present the first deep-learning framework for exemplar-based 3D model recolor, which can automatically transfer the colors from a reference image to the 3D model texture. Our framework consists of two modules to solve two major challenges in the 3D color transfer. First, we propose a new feed-forward Color Transfer Network to achieve high-quality semantic-level color transfer by finding dense semantic correspondences between images. Second, considering 3D model constraints such as UV mapping, we design a novel 3D Texture Optimization Module which can generate a seamless and coherent texture by combining color transferred results rendered in multiple views. Experiments show that our method performs robustly and generalizes well to various kinds of models.
- Open Access
AutoRemover: Automatic Object Removal for Autonomous Driving Videos
Motivated by the need for photo-realistic simulation in autonomous driving, in this paper we present a video inpainting algorithm \emph{AutoRemover}, designed specifically for generating street-view videos without any moving objects. In our setup we have two challenges: the first is the shadow, shadows are usually unlabeled but tightly coupled with the moving objects. The second is the large ego-motion in the videos. To deal with shadows, we build up an autonomous driving shadow dataset and design a deep neural network to detect shadows automatically. To deal with large ego-motion, we take advantage of the multi-source data, in particular the 3D data, in autonomous driving. More specifically, the geometric relationship between frames is incorporated into an inpainting deep neural network to produce high-quality structurally consistent video output. Experiments show that our method outperforms other state-of-the-art (SOTA) object removal algorithms, reducing the RMSE by over $19\%$.Open AccessAutoRemover: Automatic Object Removal for Autonomous Driving Videos
Motivated by the need for photo-realistic simulation in autonomous driving, in this paper we present a video inpainting algorithm \emph{AutoRemover}, designed specifically for generating street-view videos without any moving objects. In our setup we have two challenges: the first is the shadow, shadows are usually unlabeled but tightly coupled with the moving objects. The second is the large ego-motion in the videos. To deal with shadows, we build up an autonomous driving shadow dataset and design a deep neural network to detect shadows automatically. To deal with large ego-motion, we take advantage of the multi-source data, in particular the 3D data, in autonomous driving. More specifically, the geometric relationship between frames is incorporated into an inpainting deep neural network to produce high-quality structurally consistent video output. Experiments show that our method outperforms other state-of-the-art (SOTA) object removal algorithms, reducing the RMSE by over $19\%$. - 22 November 2019
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 34 (6) , 1294-1306
Artistic Augmentation of Photographs with Droplets
- 22 November 2019
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 34 (6) , 1294-1306
Artistic Augmentation of Photographs with Droplets
- 29 October 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 20 (9) , 3375-3385
A User Study on the Capability of Three Geo-Based Features in Analyzing and Locating Trajectories
Visual analysis is widely applied to study human mobility due to the ability of integrating contextual information from multiple data sources. Analyzing trajectory data through visualization improves the efficiency and accuracy of the analysis, yet it may induce exposure of the location privacy. To balance the location privacy and analysis effectiveness, this work focuses on the behaviors of different geo-based contexts in the process of trajectory interpretation. Three types of geo-based contexts are identified after surveying 94 related literatures. We further conduct experiments to investigate their capability by evaluating how they benefit the analysis, and whether they lead to location privacy exposure. Finally, we report and discuss interesting findings, and provide guidelines to the design of privacy-preserving analysis approaches for human periodic trajectories.
- 29 October 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 20 (9) , 3375-3385
A User Study on the Capability of Three Geo-Based Features in Analyzing and Locating Trajectories
Visual analysis is widely applied to study human mobility due to the ability of integrating contextual information from multiple data sources. Analyzing trajectory data through visualization improves the efficiency and accuracy of the analysis, yet it may induce exposure of the location privacy. To balance the location privacy and analysis effectiveness, this work focuses on the behaviors of different geo-based contexts in the process of trajectory interpretation. Three types of geo-based contexts are identified after surveying 94 related literatures. We further conduct experiments to investigate their capability by evaluating how they benefit the analysis, and whether they lead to location privacy exposure. Finally, we report and discuss interesting findings, and provide guidelines to the design of privacy-preserving analysis approaches for human periodic trajectories.
- Open Access
- 5 September 2018
- Published by Springer Nature in Visual Computing for Industry, Biomedicine, and Art
- Vol. 1 (1) , 1-19
Computational aesthetics and applications
Computational aesthetics, which bridges science and art, is emerging as a new interdisciplinary field. This paper concentrates on two main aspects of computational aesthetics: aesthetic measurement and quantification, generative art, and then proposes a design generation framework. On aesthetic measurement and quantification, we review different types of features used in measurement, the currently used evaluation methods, and their applications. On generative art, we focus on both fractal art and abstract paintings modeled on well-known artists’ styles. In general, computational aesthetics exploits computational methods for aesthetic expressions. In other words, it enables computer to appraise beauty and ugliness and also automatically generate aesthetic images. Computational aesthetics has been widely applied to many areas, such as photography, fine art, Chinese hand-writing, web design, graphic design, and industrial design. We finally propose a design generation methodology, utilizing techniques from both aesthetic measurements and generative art.
- 5 September 2018
- Published by Springer Nature in Visual Computing for Industry, Biomedicine, and Art
- Vol. 1 (1) , 1-19
Open AccessComputational aesthetics and applications
Computational aesthetics, which bridges science and art, is emerging as a new interdisciplinary field. This paper concentrates on two main aspects of computational aesthetics: aesthetic measurement and quantification, generative art, and then proposes a design generation framework. On aesthetic measurement and quantification, we review different types of features used in measurement, the currently used evaluation methods, and their applications. On generative art, we focus on both fractal art and abstract paintings modeled on well-known artists’ styles. In general, computational aesthetics exploits computational methods for aesthetic expressions. In other words, it enables computer to appraise beauty and ugliness and also automatically generate aesthetic images. Computational aesthetics has been widely applied to many areas, such as photography, fine art, Chinese hand-writing, web design, graphic design, and industrial design. We finally propose a design generation methodology, utilizing techniques from both aesthetic measurements and generative art.