Personalized Image Aesthetics Assessment via Meta-Learning With Bilevel Gradient Optimization
- 11 June 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 52 (3) , 1798-1811
Abstract
Typical image aesthetics assessment (IAA) is modeled for the generic aesthetics perceived by an “average” user. However, such generic aesthetics models neglect the fact that users’ aesthetic preferences vary significantly depending on their unique preferences. Therefore, it is essential to tackle the issue for personalized IAA (PIAA). Since PIAA is a typical small sample learning (SSL) problem, existing PIAA models are usually built by fine-tuning the well-established generic IAA (GIAA) models, which are regarded as prior knowledge. Nevertheless, this kind of prior knowledge based on “average aesthetics” fails to incarnate the aesthetic diversity of different people. In order to learn the shared prior knowledge when different people judge aesthetics, that is, learn how people judge image aesthetics, we propose a PIAA method based on meta-learning with bilevel gradient optimization (BLG-PIAA), which is trained using individual aesthetic data directly and generalizes to unknown users quickly. The proposed approach consists of two phases: 1) meta-training and 2) meta-testing. In meta-training, the aesthetics assessment of each user is regarded as a task, and the training set of each task is divided into two sets: 1) support set and 2) query set. Unlike traditional methods that train a GIAA model based on average aesthetics, we train an aesthetic meta-learner model by bilevel gradient updating from the support set to the query set using many users’ PIAA tasks. In meta-testing, the aesthetic meta-learner model is fine-tuned using a small amount of aesthetic data of a target user to obtain the PIAA model. The experimental results show that the proposed method outperforms the state-of-the-art PIAA metrics, and the learned prior model of BLG-PIAA can be quickly adapted to unseen PIAA tasks.Keywords
Funding Information
- Natural Science Foundation of Jiangsu Province (BK20181354)
- Science and Technology Plan of Xi’an (20191122015KYPT011JC013)
- Postgraduate Research and Practice Innovation Program of Jiangsu Province (KYCX19_2142)
- Postgraduate Research and Practice Innovation Program of China University of Mining and Technology (KYCX19_2142)
- National Natural Science Foundation of China (61771473, 61991451, 61379143)
- Six Talent Peaks High-Level Talents in Jiangsu Province (XYDXX-063)
This publication has 42 references indexed in Scilit:
- ‘Who Likes What and, Why?’ Insights into Modeling Users’ Personality Based on Image ‘Likes’IEEE Transactions on Affective Computing, 2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Unified Photo Enhancement by Discovering Aesthetic Communities From FlickrIEEE Transactions on Image Processing, 2016
- The Analysis of Image Contrast: From Quality Assessment to Automatic EnhancementIEEE Transactions on Cybernetics, 2015
- Data-Driven Affective Filtering for Images and VideosIEEE Transactions on Cybernetics, 2015
- Collaborative filtering of color aesthetics2014
- AVA: A large-scale database for aesthetic visual analysis2012
- An overview of bilevel optimizationAnnals of Operations Research, 2007
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005
- Visual perception of numerosity in infancy.Developmental Psychology, 1990