Bo Ren
Research Subjects
Bo Ren has a total of 37 co-authors. Here are the top co-authors by number of shared publications:
- Ruizhi Qiao
Tencent Youtu Lab
4 shared publications - Xiujun Shu
Peng Cheng Laboratory, Shenzhen, China
3 shared publications - Bo Ke
Tencent Youtu Lab, Shenzhen, China
2 shared publications - Jingjing Wu
Hefei University of Technology, Hefei, China
2 shared publications - Meibin Qi
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
2 shared publications - Bin Wang
Tencent Youtu Lab, China
1 shared publication
Latest Publications
- Open Access
- 4 December 2023
- Published by Association for Computing Machinery (ACM)
- p. 609-620
DeepContract: Controllable Authorization of Deep Learning Models
Well-trained deep learning (DL) models are widely used in various fields and recognized as valuable intellectual property. However, most existing efforts to fully exploit their value either require users to upload input data to provide machine learning services, which raises serious privacy concerns, or deploy DL models on the user side, resulting in a loss of control over the models. While a few active model authorization methods protect the model from unauthorized users, they cannot prevent the model from being redistributed or abused by authorized users. To address the urgent need to efficiently protect both model confidentiality and input data privacy, and achieve uninterrupted model controllability, we propose a contract-based model authorization framework called DeepContract. This framework enables model owners to deploy their models on the user side for local inference without revealing original model weights. Moreover, it allows them to grant and revoke the right to use their models at any time. Specifically, we propose a generic model encryption method that significantly outperforms the state-of-the-art method in both efficiency and security. Leveraging the integrity verification in the Trusted Execution Environment, contract-based and verifiable enclave codes generated by DeepContract can perform controlled inference using the encrypted model distributed on the user side. Our extensive evaluations show that DeepContract can achieve efficient and secure controllable model authorization for the pre-signed contract.
- 4 December 2023
- Published by Association for Computing Machinery (ACM)
- p. 609-620
Open AccessDeepContract: Controllable Authorization of Deep Learning Models
Well-trained deep learning (DL) models are widely used in various fields and recognized as valuable intellectual property. However, most existing efforts to fully exploit their value either require users to upload input data to provide machine learning services, which raises serious privacy concerns, or deploy DL models on the user side, resulting in a loss of control over the models. While a few active model authorization methods protect the model from unauthorized users, they cannot prevent the model from being redistributed or abused by authorized users. To address the urgent need to efficiently protect both model confidentiality and input data privacy, and achieve uninterrupted model controllability, we propose a contract-based model authorization framework called DeepContract. This framework enables model owners to deploy their models on the user side for local inference without revealing original model weights. Moreover, it allows them to grant and revoke the right to use their models at any time. Specifically, we propose a generic model encryption method that significantly outperforms the state-of-the-art method in both efficiency and security. Leveraging the integrity verification in the Trusted Execution Environment, contract-based and verifiable enclave codes generated by DeepContract can perform controlled inference using the encrypted model distributed on the user side. Our extensive evaluations show that DeepContract can achieve efficient and secure controllable model authorization for the pre-signed contract.
- 19 August 2023
- Published by Springer Nature
- p. 36-53
Visual Information Extraction in the Wild: Practical Dataset and End-to-End Solution
- 19 August 2023
- Published by Springer Nature
- p. 36-53
Visual Information Extraction in the Wild: Practical Dataset and End-to-End Solution
- Open Access
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (1) , 808-816
Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer
Real-world recognition system often encounters the challenge of unseen labels. To identify such unseen labels, multi-label zero-shot learning (ML-ZSL) focuses on transferring knowledge by a pre-trained textual label embedding (e.g., GloVe). However, such methods only exploit single-modal knowledge from a language model, while ignoring the rich semantic information inherent in image-text pairs. Instead, recently developed open-vocabulary (OV) based methods succeed in exploiting such information of image-text pairs in object detection, and achieve impressive performance. Inspired by the success of OV-based methods, we propose a novel open-vocabulary framework, named multi-modal knowledge transfer (MKT), for multi-label classification. Specifically, our method exploits multi-modal knowledge of image-text pairs based on a vision and language pre-training (VLP) model. To facilitate transferring the image-text matching ability of VLP model, knowledge distillation is employed to guarantee the consistency of image and label embeddings, along with prompt tuning to further update the label embeddings. To further enable the recognition of multiple objects, a simple but effective two-stream module is developed to capture both local and global features. Extensive experimental results show that our method significantly outperforms state-of-the-art methods on public benchmark datasets.
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (1) , 808-816
Open AccessOpen-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer
Real-world recognition system often encounters the challenge of unseen labels. To identify such unseen labels, multi-label zero-shot learning (ML-ZSL) focuses on transferring knowledge by a pre-trained textual label embedding (e.g., GloVe). However, such methods only exploit single-modal knowledge from a language model, while ignoring the rich semantic information inherent in image-text pairs. Instead, recently developed open-vocabulary (OV) based methods succeed in exploiting such information of image-text pairs in object detection, and achieve impressive performance. Inspired by the success of OV-based methods, we propose a novel open-vocabulary framework, named multi-modal knowledge transfer (MKT), for multi-label classification. Specifically, our method exploits multi-modal knowledge of image-text pairs based on a vision and language pre-training (VLP) model. To facilitate transferring the image-text matching ability of VLP model, knowledge distillation is employed to guarantee the consistency of image and label embeddings, along with prompt tuning to further update the label embeddings. To further enable the recognition of multiple objects, a simple but effective two-stream module is developed to capture both local and global features. Extensive experimental results show that our method significantly outperforms state-of-the-art methods on public benchmark datasets.
- Top Cited Papers
- 18 February 2023
- Published by Springer Nature
- p. 624-641
See Finer, See More: Implicit Modality Alignment for Text-Based Person Retrieval
- 18 February 2023
- Published by Springer Nature
- p. 624-641
Top Cited PapersSee Finer, See More: Implicit Modality Alignment for Text-Based Person Retrieval
- 20 October 2022
- Published by Springer Nature
- p. 38-55
Hyperspherical Learning in Multi-Label Classification
- 20 October 2022
- Published by Springer Nature
- p. 38-55
Hyperspherical Learning in Multi-Label Classification
- Open Access
Head and Body: Unified Detector and Graph Network for Person Search in Media
Person search in media has seen increasing potential in Internet applications, such as video clipping and character collection. This task is common but overlooked by previous person search works which focus on surveillance scenes. The media scenarios have some different challenges from surveillance scenes. For example, a person may change his clothes frequently. To alleviate this issue, this paper proposes a Unified Detector and Graph Network (UDGNet) for person search in media. UDGNet is the first person search framework to detect and re-identify the human body and head simultaneously. Specifically, it first builds two branches based on a unified network to detect the human body and head, then the detected body and head are used for re-identification. This dual-task approach can significantly enhance discriminative learning. To tackle the cloth-changing issue, UDGNet builds two graphs to explore reliable links among cloth-changing samples and utilizes a graph network to learn better embeddings. This design effectively enhances the robustness of person search to cloth-changing challenges. Besides, we demonstrate that UDGNet can be implemented with both anchor-based and anchor-free person search frameworks and further achieve performance improvement. This paper also contributes a large-scale dataset for Person Search in Media (PSM), which provides both body and head annotations. It is by far the largest dataset for person search in media. Experiments show that UDGNet improves the anchor-free model AlignPS by 12.1% in mAP. Meanwhile, it shows good generalization across surveillance and longterm scenarios. The dataset and code will be available at: https://github.com/shuxjweb/PSM.git.
Open AccessHead and Body: Unified Detector and Graph Network for Person Search in Media
Person search in media has seen increasing potential in Internet applications, such as video clipping and character collection. This task is common but overlooked by previous person search works which focus on surveillance scenes. The media scenarios have some different challenges from surveillance scenes. For example, a person may change his clothes frequently. To alleviate this issue, this paper proposes a Unified Detector and Graph Network (UDGNet) for person search in media. UDGNet is the first person search framework to detect and re-identify the human body and head simultaneously. Specifically, it first builds two branches based on a unified network to detect the human body and head, then the detected body and head are used for re-identification. This dual-task approach can significantly enhance discriminative learning. To tackle the cloth-changing issue, UDGNet builds two graphs to explore reliable links among cloth-changing samples and utilizes a graph network to learn better embeddings. This design effectively enhances the robustness of person search to cloth-changing challenges. Besides, we demonstrate that UDGNet can be implemented with both anchor-based and anchor-free person search frameworks and further achieve performance improvement. This paper also contributes a large-scale dataset for Person Search in Media (PSM), which provides both body and head annotations. It is by far the largest dataset for person search in media. Experiments show that UDGNet improves the anchor-free model AlignPS by 12.1% in mAP. Meanwhile, it shows good generalization across surveillance and longterm scenarios. The dataset and code will be available at: https://github.com/shuxjweb/PSM.git.
- 24 March 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 30 (10) , 3398-3412
Person Attribute Recognition by Sequence Contextual Relation Learning
Person attribute recognition aims to identify the attribute labels from the pedestrian images. Extracting contextual relation from the images and attributes, including the spatial-semantic relations, the spatial context and the semantic correlation, is beneficial to enhance the discrimination of the features for recognizing the attributes. Thus, this work proposes a sequence contextual relation learning (SCRL) method to capture these relations. It first embeds the images and attributes into sequences in two branches. Then SCRL flexibly learns the contextual relation from the sequences with the parallel attention model structure, which integrates the inter-attention and intra-attention models. The inter-attention module is utilized to extract the spatial-semantic relations, while the intra-attention is designed to gain the spatial context and the semantic correlation. Both attention modules are comprised of several parallel attention units and each unit can obtain the pairwise relations in one subspace. Therefore, they obtain the relations in multiple subspaces, which can improve the comprehensiveness of the relation learning. Additionally, for the sake of better extraction of spatial-semantic relations, this paper employs connectionist temporal classification (CTC) loss which is capable of driving the network to enforce monotonic alignment between the image and attribute. It can also accelerate the convergence of the network by the algorithm in it. Extensive experiments on five public datasets, i.e., Market-1501 attribute, Duke attribute, PETA, RAP and PA-100K datasets, demonstrate the effectiveness of the proposed method.
- 24 March 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 30 (10) , 3398-3412
Person Attribute Recognition by Sequence Contextual Relation Learning
Person attribute recognition aims to identify the attribute labels from the pedestrian images. Extracting contextual relation from the images and attributes, including the spatial-semantic relations, the spatial context and the semantic correlation, is beneficial to enhance the discrimination of the features for recognizing the attributes. Thus, this work proposes a sequence contextual relation learning (SCRL) method to capture these relations. It first embeds the images and attributes into sequences in two branches. Then SCRL flexibly learns the contextual relation from the sequences with the parallel attention model structure, which integrates the inter-attention and intra-attention models. The inter-attention module is utilized to extract the spatial-semantic relations, while the intra-attention is designed to gain the spatial context and the semantic correlation. Both attention modules are comprised of several parallel attention units and each unit can obtain the pairwise relations in one subspace. Therefore, they obtain the relations in multiple subspaces, which can improve the comprehensiveness of the relation learning. Additionally, for the sake of better extraction of spatial-semantic relations, this paper employs connectionist temporal classification (CTC) loss which is capable of driving the network to enforce monotonic alignment between the image and attribute. It can also accelerate the convergence of the network by the algorithm in it. Extensive experiments on five public datasets, i.e., Market-1501 attribute, Duke attribute, PETA, RAP and PA-100K datasets, demonstrate the effectiveness of the proposed method.
- Open Access
Sequence-based Person Attribute Recognition with Joint CTC-Attention Model
Attribute recognition has become crucial because of its wide applications in many computer vision tasks, such as person re-identification. Like many object recognition problems, variations in viewpoints, illumination, and recognition at far distance, all make this task challenging. In this work, we propose a joint CTC-Attention model (JCM), which maps attribute labels into sequences to learn the semantic relationship among attributes. Besides, this network uses neural network to encode images into sequences, and employs connectionist temporal classification (CTC) loss to train the network with the aim of improving the encoding performance of the network. At the same time, it adopts the attention model to decode the sequences, which can realize aligning the sequences and better learning the semantic information from attributes. Extensive experiments on three public datasets, i.e., Market-1501 attribute dataset, Duke attribute dataset and PETA dataset, demonstrate the effectiveness of the proposed method.
Open AccessSequence-based Person Attribute Recognition with Joint CTC-Attention Model
Attribute recognition has become crucial because of its wide applications in many computer vision tasks, such as person re-identification. Like many object recognition problems, variations in viewpoints, illumination, and recognition at far distance, all make this task challenging. In this work, we propose a joint CTC-Attention model (JCM), which maps attribute labels into sequences to learn the semantic relationship among attributes. Besides, this network uses neural network to encode images into sequences, and employs connectionist temporal classification (CTC) loss to train the network with the aim of improving the encoding performance of the network. At the same time, it adopts the attention model to decode the sequences, which can realize aligning the sequences and better learning the semantic information from attributes. Extensive experiments on three public datasets, i.e., Market-1501 attribute dataset, Duke attribute dataset and PETA dataset, demonstrate the effectiveness of the proposed method.