Peiyin Xing
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Peiyin Xing has a total of 24 co-authors. Here are the top co-authors by number of shared publications:
Yonghong Tian
Peking University, China
6 shared publicationsYaowei Wang
Pengcheng Laboratory, China
5 shared publications- Tiejun Huang
Peking University
4 shared publications - Yonghong Tian
Peking University
4 shared publications Tiejun Huang
Peking University, China
3 shared publicationsWen Gao
School of Electronics Engineering and Computer Science, Peking University, Beijing, China
3 shared publications
Latest Publications
- 22 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (4) , 3732-3744
High Efficient 3D Convolution Feature Compression
In this paper, a high efficient 3D convolution feature compression method is proposed. This method is mainly used to compress the three-dimensional convolution deep features extracted by video analysis. Gather the compressed features to the cloud server instead of all video features. This method can solve the problem that the data aggregation of video big data analysis requires a large amount of network bandwidth. Quantization is a common method for feature compression, but the existing quantization-based methods often carry out model training and quantization in stages, which makes the robustness of quantization results poor. To solve this problem, the method proposed in this paper is to apply the feature quantization operation directly to the network and train it with the analysis task, and use the parameter iterative optimization method to solve the non-differentiable problem of quantization operation. Different from the deep features extracted from images or single objects, the 3D convolution features extracted from video clips have high time-domain redundancy. In this paper, by serializing the three-dimensional convolution features, the time-domain prediction coding method is used to remove the time-domain redundancy of the three-dimensional convolution features, to improve the feature compression ratio. The experimental results show that this method can only use 1 bit to represent the elements in the three-dimensional convolution deep feature. When the analysis accuracy loss is no more than 1%, the feature compression ratio can reach 4500 times compared with the original feature data, and the data transmission can be reduced by 96%.
- 22 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (4) , 3732-3744
High Efficient 3D Convolution Feature Compression
In this paper, a high efficient 3D convolution feature compression method is proposed. This method is mainly used to compress the three-dimensional convolution deep features extracted by video analysis. Gather the compressed features to the cloud server instead of all video features. This method can solve the problem that the data aggregation of video big data analysis requires a large amount of network bandwidth. Quantization is a common method for feature compression, but the existing quantization-based methods often carry out model training and quantization in stages, which makes the robustness of quantization results poor. To solve this problem, the method proposed in this paper is to apply the feature quantization operation directly to the network and train it with the analysis task, and use the parameter iterative optimization method to solve the non-differentiable problem of quantization operation. Different from the deep features extracted from images or single objects, the 3D convolution features extracted from video clips have high time-domain redundancy. In this paper, by serializing the three-dimensional convolution features, the time-domain prediction coding method is used to remove the time-domain redundancy of the three-dimensional convolution features, to improve the feature compression ratio. The experimental results show that this method can only use 1 bit to represent the elements in the three-dimensional convolution deep feature. When the analysis accuracy loss is no more than 1%, the feature compression ratio can reach 4500 times compared with the original feature data, and the data transmission can be reduced by 96%.
- 16 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 31 (11) , 4147-4161
Digital Retina: A Way to Make the City Brain More Efficient by Visual Coding
The ubiquitous camera networks in the city brain system grow at a rapid pace, creating massive amounts of images and videos at a range of spatial-temporal scales and thereby forming the “biggest” big data. However, the sensing system often lags behind the construction of the fast-growing city brain system, in the sense that such exponentially growing data far exceed today’s sensing capabilities. Therefore, critical issues arise regarding how to better leverage the existing city brain system and significantly improve the city-scale performance in intelligent applications. To tackle the unprecedented challenges, we articulate a vision towards a novel visual computing framework, termed as digital retina , which aligns high-efficiency sensing models with the emerging Visual Coding for Machine (VCM) paradigm. In particular, digital retina may consist of video coding, feature coding, model coding, as well as their joint optimization. The digital retina is biologically-inspired, rooted on the widely accepted view that the retina encodes the visual information for human perception, and extracts features by the brain downstream areas to disentangle the visual objects. Within the digital retina framework, three streams, i.e., video stream, feature stream, and model stream, work collaboratively over the end-edge-cloud platform. In particular, the compressed video stream serves for human vision, the compact feature stream targets for machine vision, and the model stream incrementally updates deep learning models to improve the performance of human/machine vision tasks. We have developed a prototype to demonstrate the technical advantages of digital retina, and extensive experiments have been conducted to validate that it is able to effectively support the video big data analysis and retrieval in the intelligent city system. In particular, up to $7000\times $ compression ratio could be realized for visual data compression while maintaining competitive performance with pristine signal in a series of visual analysis tasks.
- 16 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 31 (11) , 4147-4161
Digital Retina: A Way to Make the City Brain More Efficient by Visual Coding
The ubiquitous camera networks in the city brain system grow at a rapid pace, creating massive amounts of images and videos at a range of spatial-temporal scales and thereby forming the “biggest” big data. However, the sensing system often lags behind the construction of the fast-growing city brain system, in the sense that such exponentially growing data far exceed today’s sensing capabilities. Therefore, critical issues arise regarding how to better leverage the existing city brain system and significantly improve the city-scale performance in intelligent applications. To tackle the unprecedented challenges, we articulate a vision towards a novel visual computing framework, termed as digital retina , which aligns high-efficiency sensing models with the emerging Visual Coding for Machine (VCM) paradigm. In particular, digital retina may consist of video coding, feature coding, model coding, as well as their joint optimization. The digital retina is biologically-inspired, rooted on the widely accepted view that the retina encodes the visual information for human perception, and extracts features by the brain downstream areas to disentangle the visual objects. Within the digital retina framework, three streams, i.e., video stream, feature stream, and model stream, work collaboratively over the end-edge-cloud platform. In particular, the compressed video stream serves for human vision, the compact feature stream targets for machine vision, and the model stream incrementally updates deep learning models to improve the performance of human/machine vision tasks. We have developed a prototype to demonstrate the technical advantages of digital retina, and extensive experiments have been conducted to validate that it is able to effectively support the video big data analysis and retrieval in the intelligent city system. In particular, up to $7000\times $ compression ratio could be realized for visual data compression while maintaining competitive performance with pristine signal in a series of visual analysis tasks.
- 6 June 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8483-8487
Allocating DNN Layers Computation Between Front-End Devices and The Cloud Server for Video Big Data Processing
With the development of intelligent hardware, front-end devices can also perform DNN computation. Moreover, the deep neural network can be divided into several layers. In this way, part of the computation of DNN models can be migrated to the front-end devices, which can alleviate the cloud burden and shorten the processing latency. This paper proposes a computation allocation algorithm of DNN between the front-end devices and the cloud server. In brief, we divide the DNN layers dynamically according to the current and the predicted future status of the processing system, by which we obtain a shorter end-to-end latency. The simulation results reveal that the overall latency reduction is more than 70% compared with traditional cloud-centered processing.
- 6 June 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8483-8487
Allocating DNN Layers Computation Between Front-End Devices and The Cloud Server for Video Big Data Processing
With the development of intelligent hardware, front-end devices can also perform DNN computation. Moreover, the deep neural network can be divided into several layers. In this way, part of the computation of DNN models can be migrated to the front-end devices, which can alleviate the cloud burden and shorten the processing latency. This paper proposes a computation allocation algorithm of DNN between the front-end devices and the cloud server. In brief, we divide the DNN layers dynamically according to the current and the predicted future status of the processing system, by which we obtain a shorter end-to-end latency. The simulation results reveal that the overall latency reduction is more than 70% compared with traditional cloud-centered processing.
- Open Access
- 15 October 2020
- Published by MDPI AG in Mathematics
- Vol. 8 (10) , 1785
Approximation Algorithms for the Submodular Load Balancing with Submodular Penalties
In this paper, we study the submodular load balancing problem with submodular penalties. The objective of this problem is to balance the load among sets, while some elements can be rejected by paying some penalties. Officially, given an element set V, we want to find a subset R of rejected elements, and assign other elements to one of m sets . The objective is to minimize the sum of the maximum load among and the rejection penalty of R, where the load and rejection penalty are determined by different submodular functions. We study the submodular load balancing problem with submodular penalties under two settings: heterogenous setting (load functions are not identical) and homogenous setting (load functions are identical). Moreover, we design a Lovász rounding algorithm achieving a worst-case guarantee of under the heterogenous setting and a -approximation combinatorial algorithm under the homogenous setting.- 15 October 2020
- Published by MDPI AG in Mathematics
- Vol. 8 (10) , 1785
Open AccessApproximation Algorithms for the Submodular Load Balancing with Submodular Penalties
In this paper, we study the submodular load balancing problem with submodular penalties. The objective of this problem is to balance the load among sets, while some elements can be rejected by paying some penalties. Officially, given an element set V, we want to find a subset R of rejected elements, and assign other elements to one of m sets . The objective is to minimize the sum of the maximum load among and the rejection penalty of R, where the load and rejection penalty are determined by different submodular functions. We study the submodular load balancing problem with submodular penalties under two settings: heterogenous setting (load functions are not identical) and homogenous setting (load functions are identical). Moreover, we design a Lovász rounding algorithm achieving a worst-case guarantee of under the heterogenous setting and a -approximation combinatorial algorithm under the homogenous setting. - 1 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 73-78
Prune it Yourself: Automated Pruning by Multiple Level Sensitivity
Deep neural network pruning is to reduce the model size by removing redundant structures and weights. Existing methods focus on single layer information which ignore other layers. And pruning progress is simply removing all weights at the same time. To address these limitations, we propose Prune it Yourself (PIY) framework. First, we collect both filter and channel sensitivity information. Then combine them to decide the structure to be pruned. At last we use the gradual pruning algorithm to reduce the accuracy loss without extra hyper-parameters. We use VGG-16 and ResNet to perform experiments on CIFAR-10 and ImageNet. The experimental results prove the effectiveness of our method.
- 1 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 73-78
Prune it Yourself: Automated Pruning by Multiple Level Sensitivity
Deep neural network pruning is to reduce the model size by removing redundant structures and weights. Existing methods focus on single layer information which ignore other layers. And pruning progress is simply removing all weights at the same time. To address these limitations, we propose Prune it Yourself (PIY) framework. First, we collect both filter and channel sensitivity information. Then combine them to decide the structure to be pruned. At last we use the gradual pruning algorithm to reduce the accuracy loss without extra hyper-parameters. We use VGG-16 and ResNet to perform experiments on CIFAR-10 and ImageNet. The experimental results prove the effectiveness of our method.
- 1 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 233-236
End-Edge-Cloud Collaborative System: A Video Big Data Processing and Analysis Architecture
In the current surveillance system, video streams are firstly captured and compressed at the cameras, and then transmitted to the backend severs or cloud for big data analysis. It is impractical to aggregate all video streams from hundreds of thousands of cameras for big data analysis. Transcoding the videos to low-bitrate ones is the conventional solution to solve the aggregation bottleneck. However, it is recognized that transcoding will inevitably affect visual feature extraction, consequently degrading the subsequent analysis performance. To address these challenges, we thus propose a new video big data analysis framework, called end-edge-cloud collaborative system. Under the end-edge-cloud collaborative framework, a camera can output two streams simultaneously, including a compressed video stream for viewing and data storage, and a compact feature stream extracted from the original video signals for visual analysis. Video stream and feature stream are synchronized by unified identification. We identify three key technologies to enable the end-edge-cloud collaborative system, including analysis-friendly video coding, visual feature compact descriptor, and user-defined neural network and parameter updating. By real-time feeding only the feature streams into the cloud center, these cameras thus form a large-scale brain-like vision system for the smart city. A prototype has been implemented to demonstrate its feasibility. Experiment results show that our system can achieve high efficient video compression and guarantee the analysis performance. Furthermore, our system makes the big data analysis feasible which only need aggregate low bit-rate com-pressed feature stream.
- 1 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 233-236
End-Edge-Cloud Collaborative System: A Video Big Data Processing and Analysis Architecture
In the current surveillance system, video streams are firstly captured and compressed at the cameras, and then transmitted to the backend severs or cloud for big data analysis. It is impractical to aggregate all video streams from hundreds of thousands of cameras for big data analysis. Transcoding the videos to low-bitrate ones is the conventional solution to solve the aggregation bottleneck. However, it is recognized that transcoding will inevitably affect visual feature extraction, consequently degrading the subsequent analysis performance. To address these challenges, we thus propose a new video big data analysis framework, called end-edge-cloud collaborative system. Under the end-edge-cloud collaborative framework, a camera can output two streams simultaneously, including a compressed video stream for viewing and data storage, and a compact feature stream extracted from the original video signals for visual analysis. Video stream and feature stream are synchronized by unified identification. We identify three key technologies to enable the end-edge-cloud collaborative system, including analysis-friendly video coding, visual feature compact descriptor, and user-defined neural network and parameter updating. By real-time feeding only the feature streams into the cloud center, these cameras thus form a large-scale brain-like vision system for the smart city. A prototype has been implemented to demonstrate its feasibility. Experiment results show that our system can achieve high efficient video compression and guarantee the analysis performance. Furthermore, our system makes the big data analysis feasible which only need aggregate low bit-rate com-pressed feature stream.
- 1 March 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 400
Binary Representation and High Efficient Compression of 3D CNN Features for Action Recognition
A common framework of the action recognition is to collect the videos from different cameras into a cloud center firstly, and then perform the 3D CNN on the cloud server. Although directly, this framework will bring a huge burden to the cloud server and video transmission. To handle this challenge, the "front-cloud" collaborative processing architecture can be used. The most import issue is to compress the feature from 3D CNN effectively without significant loss of accuracy. We propose logarithmic quantization with a maximum value threshold and HEVC inter encoding for 3D CNN features. Experimental results on ResNet-50 and InceptionV1 show that the features can be represented by only 1 bit without significant loss of accuracy. The compression ratio of the quantized 1 bit features using HEVC inter coding can reach to 5000 times and the loss of accuracy is less than 1%.
- 1 March 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 400
Binary Representation and High Efficient Compression of 3D CNN Features for Action Recognition
A common framework of the action recognition is to collect the videos from different cameras into a cloud center firstly, and then perform the 3D CNN on the cloud server. Although directly, this framework will bring a huge burden to the cloud server and video transmission. To handle this challenge, the "front-cloud" collaborative processing architecture can be used. The most import issue is to compress the feature from 3D CNN effectively without significant loss of accuracy. We propose logarithmic quantization with a maximum value threshold and HEVC inter encoding for 3D CNN features. Experimental results on ResNet-50 and InceptionV1 show that the features can be represented by only 1 bit without significant loss of accuracy. The compression ratio of the quantized 1 bit features using HEVC inter coding can reach to 5000 times and the loss of accuracy is less than 1%.
- 12 June 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE MultiMedia
- Vol. 26 (4) , 67-79
Residual-Based Post-Processing for HEVC
We proposed a residual-based post-processing for High Efficiency Video Coding (HEVC). Based on the proposed network, Residual-based Video Restoration Network (Residual-VRN), the decoded image quality has been improved significantly. We experimentally verified the superiority of the method.
- 12 June 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE MultiMedia
- Vol. 26 (4) , 67-79
Residual-Based Post-Processing for HEVC
We proposed a residual-based post-processing for High Efficiency Video Coding (HEVC). Based on the proposed network, Residual-based Video Restoration Network (Residual-VRN), the decoded image quality has been improved significantly. We experimentally verified the superiority of the method.
- 1 March 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 609
Efficient and Fast Coefficient Sign Inference for Video Coding
In the modern video coding standards, transform coefficient coding takes large portion of output bit-stream. The absolute value of coefficients are usually represented with high-efficiency variable-length code, while the signs are binarized with 1-bit fixed-length code, which ignores the apriority constraints on sign combinations beyond signs. In this paper, we present a fast and efficient sign inference algorithm for improving the coding efficiency of coefficients signs. By considering the correlation of pixels across the transform block boundary, we define a criterion of continuity to constrain the combination space of signs in the block, and shorten the bits will be used. A given number of signs are hidden during bypass coding, and inferred with the high-precision criterion. Then the infer-error is coded with arithmetic coding. Meanwhile a sign bit redefining strategy is introduced, which is based on the invariants of the linear orthogonal separable transform. The strategy ensures that the codec is able to choose up to 4 bits for inferring each block, and avoids the complex decision mechanism introduced by rate distortion optimization. Experiment results show that about 0.6% in average and up to 1.1% BD-rate saving is archived and only less than 1% encoding time complexity increased compared with HEVC baseline.
- 1 March 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 609
Efficient and Fast Coefficient Sign Inference for Video Coding
In the modern video coding standards, transform coefficient coding takes large portion of output bit-stream. The absolute value of coefficients are usually represented with high-efficiency variable-length code, while the signs are binarized with 1-bit fixed-length code, which ignores the apriority constraints on sign combinations beyond signs. In this paper, we present a fast and efficient sign inference algorithm for improving the coding efficiency of coefficients signs. By considering the correlation of pixels across the transform block boundary, we define a criterion of continuity to constrain the combination space of signs in the block, and shorten the bits will be used. A given number of signs are hidden during bypass coding, and inferred with the high-precision criterion. Then the infer-error is coded with arithmetic coding. Meanwhile a sign bit redefining strategy is introduced, which is based on the invariants of the linear orthogonal separable transform. The strategy ensures that the codec is able to choose up to 4 bits for inferring each block, and avoids the complex decision mechanism introduced by rate distortion optimization. Experiment results show that about 0.6% in average and up to 1.1% BD-rate saving is archived and only less than 1% encoding time complexity increased compared with HEVC baseline.
- 1 May 2015
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 304 (02714302) , 2764-2767
Quality-progressive coding for high bit-rate background frames on surveillance videos
A remarkable compress performance was achieved in the surveillance video coding when high-quality reconstructed background frames were referenced in long-term. However, the high-quality reconstructed background frames lead to bursting bit-rate peaks in the video transmission, which may cause obvious communication delay or buffer overflow. In order to address this problem, the paper proposes a quality-progressive coding algorithm for smoothing the bursting peaks caused by high-quality reconstructed background frames. Instead of a single high-quality reconstructed background frame, we divide the background frame into a set of frames, which include the basic reconstructed background frame of normal-quality (basic part) and a series of reconstructed residual frames (residual part), for transmission. Moreover, the modeled background frame and the residual frames should be encoded into the bit-stream and transmitted every several frames, and the coding bits of two frame types above should be limited to the target range of coding bits, which is based on the channel capacity, so that avoids the bursting bit-rate peak and the transmission delay. Background frames are reconstructed by summing up the basic part and reconstructed residual frames one by one, and the last reconstructed background frame becomes a high-quality reconstructed background frame. Of course, each reconstructed background frame act as a prediction reference for its subsequent frames. Experimental results on an opening dataset, PKU-SVD-A, show that the proposed approach can smooth the bit-rate of high-quality reconstructed background frame in surveillance video coding, and achieve 0.57% bit-rate saving on average compared with HEVC-S.
- 1 May 2015
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 304 (02714302) , 2764-2767
Quality-progressive coding for high bit-rate background frames on surveillance videos
A remarkable compress performance was achieved in the surveillance video coding when high-quality reconstructed background frames were referenced in long-term. However, the high-quality reconstructed background frames lead to bursting bit-rate peaks in the video transmission, which may cause obvious communication delay or buffer overflow. In order to address this problem, the paper proposes a quality-progressive coding algorithm for smoothing the bursting peaks caused by high-quality reconstructed background frames. Instead of a single high-quality reconstructed background frame, we divide the background frame into a set of frames, which include the basic reconstructed background frame of normal-quality (basic part) and a series of reconstructed residual frames (residual part), for transmission. Moreover, the modeled background frame and the residual frames should be encoded into the bit-stream and transmitted every several frames, and the coding bits of two frame types above should be limited to the target range of coding bits, which is based on the channel capacity, so that avoids the bursting bit-rate peak and the transmission delay. Background frames are reconstructed by summing up the basic part and reconstructed residual frames one by one, and the last reconstructed background frame becomes a high-quality reconstructed background frame. Of course, each reconstructed background frame act as a prediction reference for its subsequent frames. Experimental results on an opening dataset, PKU-SVD-A, show that the proposed approach can smooth the bit-rate of high-quality reconstructed background frame in surveillance video coding, and achieve 0.57% bit-rate saving on average compared with HEVC-S.