Anton V. Trusov
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Anton V. Trusov has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Elena Limonova
Smart Engines Service LLC, Moscow, Russia
7 shared publications - Vladimir V. Arlazarov
Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences, Moscow, Russian Federation
6 shared publications - Dmitrii P. Nikolaev
Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences, Moscow 119333, Russia
5 shared publications - Alexander Zatsarinnyy
Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences, Moscow, Russia
1 shared publication - Artem Sher
Moscow Institute of Physics and Technology, Dolgoprudnii, Russia
1 shared publication D.N. Putintsev
Russian Acad Sci, Fed Res Ctr Comp Sci & Control, Vavilova Str 44,Corp 2, Moscow 19333, Russia
1 shared publication
Latest Publications
- 1 March 2025
- Published by Pleiades Publishing Ltd in Pattern Recognition and Image Analysis
- Vol. 35 (1) , 44-64
Training 4.6-Bit Convolutional Neural Networks with a HardTanh Activation Function
Low-bit quantization of neural networks is of great practical importance since it helps to significantly reduce the memory footprint and power consumption as well as increase the computational speed, which is especially important for mobile devices. Acting as a compromise between accurate 8-bit quantization and computationally efficient 4-bit quantization, 4.6-bit quantization is one of the promising methods. In this work, theoretical and practical aspects of training 4.6-bit convolutional neural networks with a HardTanh-type activation function are studied. Applying such activations, one can combine quantization and activation operations, simplifying the training procedure and reducing the computational cost of execution. Combinations of three different initialization strategies and four quantization algorithms are theoretically and experimentally studied for the class of neural networks involved. Special attention is paid to quantizing blocks with residual connections. According to the results, the best accuracy can be achieved by using initialization that balances the variance of gradients in the neural network with a layer-by-layer quantization algorithm that calibrates the weights of the neural network layers similar to the AdaQuant algorithm. For quantization of a block with a residual connection, an extended HardTanh-type activation function should be used, which is subsequently combined with the quantization operation.
- 1 March 2025
- Published by Pleiades Publishing Ltd in Pattern Recognition and Image Analysis
- Vol. 35 (1) , 44-64
Training 4.6-Bit Convolutional Neural Networks with a HardTanh Activation Function
Low-bit quantization of neural networks is of great practical importance since it helps to significantly reduce the memory footprint and power consumption as well as increase the computational speed, which is especially important for mobile devices. Acting as a compromise between accurate 8-bit quantization and computationally efficient 4-bit quantization, 4.6-bit quantization is one of the promising methods. In this work, theoretical and practical aspects of training 4.6-bit convolutional neural networks with a HardTanh-type activation function are studied. Applying such activations, one can combine quantization and activation operations, simplifying the training procedure and reducing the computational cost of execution. Combinations of three different initialization strategies and four quantization algorithms are theoretically and experimentally studied for the class of neural networks involved. Special attention is paid to quantizing blocks with residual connections. According to the results, the best accuracy can be achieved by using initialization that balances the variance of gradients in the neural network with a layer-by-layer quantization algorithm that calibrates the weights of the neural network layers similar to the AdaQuant algorithm. For quantization of a block with a residual connection, an extended HardTanh-type activation function should be used, which is subsequently combined with the quantization operation.
- 1 August 2024
- Published by Samara National Research University in Computer Optics
- Vol. 48 (4) , 573-581
Uncertainty-based quantization method for stable training of binary neural networks
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs. To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.
- 1 August 2024
- Published by Samara National Research University in Computer Optics
- Vol. 48 (4) , 573-581
Uncertainty-based quantization method for stable training of binary neural networks
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs. To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.
- Open Access
- 23 February 2024
- Published by MDPI AG in Mathematics
- Vol. 12 (5) , 651
4.6-Bit Quantization for Fast and Accurate Neural Network Inference on CPUs
Quantization is a widespread method for reducing the inference time of neural networks on mobile Central Processing Units (CPUs). Eight-bit quantized networks demonstrate similarly high quality as full precision models and perfectly fit the hardware architecture with one-byte coefficients and thirty-two-bit dot product accumulators. Lower precision quantizations usually suffer from noticeable quality loss and require specific computational algorithms to outperform eight-bit quantization. In this paper, we propose a novel 4.6-bit quantization scheme that allows for more efficient use of CPU resources. This scheme has more quantization bins than four-bit quantization and is more accurate while preserving the computational efficiency of the later (it runs only 4% slower). Our multiplication uses a combination of 16- and 32-bit accumulators and avoids multiplication depth limitation, which the previous 4-bit multiplication algorithm had. The experiments with different convolutional neural networks on CIFAR-10 and ImageNet datasets show that 4.6-bit quantized networks are 1.5–1.6 times faster than eight-bit networks on the ARMv8 CPU. Regarding the quality, the results of the 4.6-bit quantized network are close to the mean of four-bit and eight-bit networks of the same architecture. Therefore, 4.6-bit quantization may serve as an intermediate solution between fast and inaccurate low-bit network quantizations and accurate but relatively slow eight-bit ones.
- 23 February 2024
- Published by MDPI AG in Mathematics
- Vol. 12 (5) , 651
Open Access4.6-Bit Quantization for Fast and Accurate Neural Network Inference on CPUs
Quantization is a widespread method for reducing the inference time of neural networks on mobile Central Processing Units (CPUs). Eight-bit quantized networks demonstrate similarly high quality as full precision models and perfectly fit the hardware architecture with one-byte coefficients and thirty-two-bit dot product accumulators. Lower precision quantizations usually suffer from noticeable quality loss and require specific computational algorithms to outperform eight-bit quantization. In this paper, we propose a novel 4.6-bit quantization scheme that allows for more efficient use of CPU resources. This scheme has more quantization bins than four-bit quantization and is more accurate while preserving the computational efficiency of the later (it runs only 4% slower). Our multiplication uses a combination of 16- and 32-bit accumulators and avoids multiplication depth limitation, which the previous 4-bit multiplication algorithm had. The experiments with different convolutional neural networks on CIFAR-10 and ImageNet datasets show that 4.6-bit quantized networks are 1.5–1.6 times faster than eight-bit networks on the ARMv8 CPU. Regarding the quality, the results of the 4.6-bit quantized network are close to the mean of four-bit and eight-bit networks of the same architecture. Therefore, 4.6-bit quantization may serve as an intermediate solution between fast and inaccurate low-bit network quantizations and accurate but relatively slow eight-bit ones.
- 1 December 2023
- Published by Pleiades Publishing Ltd in Pattern Recognition and Image Analysis
- Vol. 33 (4) , 756-768
On Fast Computing of Neural Networks Using Central Processing Units
This work is devoted to methods for creating fast and accurate neural network algorithms for central processors, which were proposed by scientists of the V.L. Arlazarov’s scientific school. It outlines general principles and approaches to improving computational efficiency and discusses specific examples: tensor convolution decompositions that simplify convolutional neural networks; bounded nonlinear activation function ratio, which is calculated faster than exponential activation functions; and p-im2col convolution algorithm, which allows you to achieve a balance between computational efficiency and RAM consumption. Particular attention is paid to quantized (8- and 4-bit integer) neural networks, their training, implementation, and limitations on some central processor architectures, such as Elbrus.
- 1 December 2023
- Published by Pleiades Publishing Ltd in Pattern Recognition and Image Analysis
- Vol. 33 (4) , 756-768
On Fast Computing of Neural Networks Using Central Processing Units
This work is devoted to methods for creating fast and accurate neural network algorithms for central processors, which were proposed by scientists of the V.L. Arlazarov’s scientific school. It outlines general principles and approaches to improving computational efficiency and discusses specific examples: tensor convolution decompositions that simplify convolutional neural networks; bounded nonlinear activation function ratio, which is calculated faster than exponential activation functions; and p-im2col convolution algorithm, which allows you to achieve a balance between computational efficiency and RAM consumption. Particular attention is paid to quantized (8- and 4-bit integer) neural networks, their training, implementation, and limitations on some central processor architectures, such as Elbrus.
- 1 December 2023
- Published by Pleiades Publishing Ltd in Programming and Computer Software
- Vol. 49 (S2) , S115-S121
Analysis of Vulnerabilities of Neural Network Image Recognition Technologies
The problem of vulnerability of artificial intelligence technologies based on neural networks is considered. It is shown that the use of neural networks generates a lot of vulnerabilities. Examples of such vulnerabilities are demonstrated, such as incorrect classification of images containing adversarial noise or patches, failure of recognition systems in the presence of special patterns in the image, including those applied to objects in the real world, training data poisoning, etc. Based on the analysis, the need to improve the security of artificial intelligence technologies is shown, and some considerations that contribute to this improvement are discussed.
- 1 December 2023
- Published by Pleiades Publishing Ltd in Programming and Computer Software
- Vol. 49 (S2) , S115-S121
Analysis of Vulnerabilities of Neural Network Image Recognition Technologies
The problem of vulnerability of artificial intelligence technologies based on neural networks is considered. It is shown that the use of neural networks generates a lot of vulnerabilities. Examples of such vulnerabilities are demonstrated, such as incorrect classification of images containing adversarial noise or patches, failure of recognition systems in the presence of special patterns in the image, including those applied to objects in the real world, training data poisoning, etc. Based on the analysis, the need to improve the security of artificial intelligence technologies is shown, and some considerations that contribute to this improvement are discussed.
- Open Access
- 3 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 82151-82166
PESAC, the Generalized Framework for RANSAC-Based Methods on SIMD Computing Platforms
This paper focuses on the computational optimization of RANSAC. We describe the Parallel Efficient Sample Consensus (PESAC) framework that allows efficient utilization of SIMD extensions and provides memory locality due to a special way of storing the input sequence of correspondences and generating a batch of samples per one main loop iteration. It is inspired by the USAC framework and has a block structure capable of implementing most modern RANSAC-based methods. We enhance it with individual blocks of sample and model restrictors that are aimed at the rejection of “bad” samples and model hypothesis before time-consuming model computation and verification blocks. We also provide a detailed description implementing 2D homography estimation problem in PESAC and benchmark the running time on the MIDV-2020 dataset of identity documents. Comparing to naive implementation, we accelerated our framework by 122 times for the document classification task (with a 6 % increase in accuracy) and by 18 times for document tracking (with a 46 % decrease in tracking failure rate) by using both restrictors and vector processing. This version also outperformed a number of USAC implementations from OpenCV-4.6.0 in runtime and accuracy of estimation (3 times faster, 6 % greater accuracy for the classification task, and 2 times faster, 33 % lower failure rate for tracking if comparing with USAC_MAGSAC).
- 3 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 82151-82166
Open AccessPESAC, the Generalized Framework for RANSAC-Based Methods on SIMD Computing Platforms
This paper focuses on the computational optimization of RANSAC. We describe the Parallel Efficient Sample Consensus (PESAC) framework that allows efficient utilization of SIMD extensions and provides memory locality due to a special way of storing the input sequence of correspondences and generating a batch of samples per one main loop iteration. It is inspired by the USAC framework and has a block structure capable of implementing most modern RANSAC-based methods. We enhance it with individual blocks of sample and model restrictors that are aimed at the rejection of “bad” samples and model hypothesis before time-consuming model computation and verification blocks. We also provide a detailed description implementing 2D homography estimation problem in PESAC and benchmark the running time on the MIDV-2020 dataset of identity documents. Comparing to naive implementation, we accelerated our framework by 122 times for the document classification task (with a 6 % increase in accuracy) and by 18 times for document tracking (with a 46 % decrease in tracking failure rate) by using both restrictors and vector processing. This version also outperformed a number of USAC implementations from OpenCV-4.6.0 in runtime and accuracy of estimation (3 times faster, 6 % greater accuracy for the classification task, and 2 times faster, 33 % lower failure rate for tracking if comparing with USAC_MAGSAC).
- Open Access
- 29 April 2023
- Published by MDPI AG in Mathematics
- Vol. 11 (9) , 2112
Neuron-by-Neuron Quantization for Efficient Low-Bit QNN Training
Quantized neural networks (QNNs) are widely used to achieve computationally efficient solutions to recognition problems. Overall, eight-bit QNNs have almost the same accuracy as full-precision networks, but working several times faster. However, the networks with lower quantization levels demonstrate inferior accuracy in comparison to their classical analogs. To solve this issue, a number of quantization-aware training (QAT) approaches were proposed. In this paper, we study QAT approaches for two- to eight-bit linear quantization schemes and propose a new combined QAT approach: neuron-by-neuron quantization with straight-through estimator (STE) gradient forwarding. It is suitable for quantizations with two- to eight-bit widths and eliminates significant accuracy drops during training, which results in better accuracy of the final QNN. We experimentally evaluate our approach on CIFAR-10 and ImageNet classification and show that it is comparable to other approaches for four to eight bits and outperforms some of them for two to three bits while being easier to implement. For example, the proposed approach to three-bit quantization of the CIFAR-10 dataset results in 73.2% accuracy, while baseline direct and layer-by-layer result in 71.4% and 67.2% accuracy, respectively. The results for two-bit quantization for ResNet18 on the ImageNet dataset are 63.69% for our approach and 61.55% for the direct baseline.
- 29 April 2023
- Published by MDPI AG in Mathematics
- Vol. 11 (9) , 2112
Open AccessNeuron-by-Neuron Quantization for Efficient Low-Bit QNN Training
Quantized neural networks (QNNs) are widely used to achieve computationally efficient solutions to recognition problems. Overall, eight-bit QNNs have almost the same accuracy as full-precision networks, but working several times faster. However, the networks with lower quantization levels demonstrate inferior accuracy in comparison to their classical analogs. To solve this issue, a number of quantization-aware training (QAT) approaches were proposed. In this paper, we study QAT approaches for two- to eight-bit linear quantization schemes and propose a new combined QAT approach: neuron-by-neuron quantization with straight-through estimator (STE) gradient forwarding. It is suitable for quantizations with two- to eight-bit widths and eliminates significant accuracy drops during training, which results in better accuracy of the final QNN. We experimentally evaluate our approach on CIFAR-10 and ImageNet classification and show that it is comparable to other approaches for four to eight bits and outperforms some of them for two to three bits while being easier to implement. For example, the proposed approach to three-bit quantization of the CIFAR-10 dataset results in 73.2% accuracy, while baseline direct and layer-by-layer result in 71.4% and 67.2% accuracy, respectively. The results for two-bit quantization for ResNet18 on the ImageNet dataset are 63.69% for our approach and 61.55% for the direct baseline.
- Open Access
- 24 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 19725-19740
Fast Computation of RFD-Like Descriptors in Four Orientations
RFD-like binary descriptors have been designed to be fast and demonstrate good quality in image-matching tasks. One of those descriptors, RFDoc, produces state-of-the art results when applied to document localization systems. However, the computational efficiency of such descriptors strongly depends on their implementation. In this study, we consider the computation of RFD-like descriptors for an 8-bit single-channel image; provide a detailed implementation of the baseline algorithm; demonstrate its weak points; and propose four modifications. In those modifications, we investigate two ways to accelerate the descriptor computations: 1) compute common operations globally for the entire input image instead of computing them locally for every patch; or 2) use lookup tables to replace the most computationally demanding operations and minimize the number of conversions between integer and floating point types. Experiments on the document identification and localization task on the MIDV-2020 dataset have shown that the modifications with lookup tables are noticeably faster than the baseline, achieving a 2-2.6 times acceleration on x86 and ARM CPUs. Based on experimental results, modifications with global operations may outperform the baseline algorithm if there are many intersecting patches that require descriptor computation. We also demonstrated that one can use any of the proposed algorithms without loss of image-matching quality and without necessity to retrain the parameters of RFD-like descriptors. Finally, we propose an efficient way to compute the descriptors for four orientations of a patch, which is a important for document location systems. The proposed method reduces a common computation part for four orientations to a single run; thus, four descriptors are computed 3 times faster than the direct computation of the descriptors for four patches, as shown experimentally.
- 24 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 19725-19740
Open AccessFast Computation of RFD-Like Descriptors in Four Orientations
RFD-like binary descriptors have been designed to be fast and demonstrate good quality in image-matching tasks. One of those descriptors, RFDoc, produces state-of-the art results when applied to document localization systems. However, the computational efficiency of such descriptors strongly depends on their implementation. In this study, we consider the computation of RFD-like descriptors for an 8-bit single-channel image; provide a detailed implementation of the baseline algorithm; demonstrate its weak points; and propose four modifications. In those modifications, we investigate two ways to accelerate the descriptor computations: 1) compute common operations globally for the entire input image instead of computing them locally for every patch; or 2) use lookup tables to replace the most computationally demanding operations and minimize the number of conversions between integer and floating point types. Experiments on the document identification and localization task on the MIDV-2020 dataset have shown that the modifications with lookup tables are noticeably faster than the baseline, achieving a 2-2.6 times acceleration on x86 and ARM CPUs. Based on experimental results, modifications with global operations may outperform the baseline algorithm if there are many intersecting patches that require descriptor computation. We also demonstrated that one can use any of the proposed algorithms without loss of image-matching quality and without necessity to retrain the parameters of RFD-like descriptors. Finally, we propose an efficient way to compute the descriptors for four orientations of a patch, which is a important for document location systems. The proposed method reduces a common computation part for four orientations to a single run; thus, four descriptors are computed 3 times faster than the direct computation of the descriptors for four patches, as shown experimentally.
- Open Access
- 14 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 168162-168184
p-im2col: Simple Yet Efficient Convolution Algorithm With Flexibly Controlled Memory Overhead
Convolution is the most time-consuming operation in modern deep artificial neural networks, so its performance is crucial for fast inference. One of the standard approaches to fast convolution computation is to use GeMM-based convolution algorithms relying on efficient general matrix multiplication (GeMM) from optimized BLAS libraries. However, commonly used GeMM-based algorithms may cause significant memory overhead or avoid it only at the cost of worse performance. In this paper, we propose a novel convolution algorithm, p-im2col, based on a well-known im2col algorithm that avoids memory overhead by splitting a single multiplication of a large matrix into several multiplications of smaller matrices. We theoretically and experimentally compare our algorithm with two other GeMM-based algorithms: im2col, which is widely used as a baseline, and the memory-efficient kn2row-aa. We measure the inference time of these algorithms on central processing units of x86, x86_64, ARM, and MIPS architectures for a large set of convolutional parameters. The proposed algorithm demonstrates a speedup over im2col and kn2row-aa in a number of cases and a significant reduction in additional memory requirements compared to im2col. Based on our experiments, we present a new convolution algorithm selection scheme that considers memory restrictions, CPU architecture, and convolutional parameters and provides a noticeable advantage over each particular algorithm.
- 14 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 168162-168184
Open Accessp-im2col: Simple Yet Efficient Convolution Algorithm With Flexibly Controlled Memory Overhead
Convolution is the most time-consuming operation in modern deep artificial neural networks, so its performance is crucial for fast inference. One of the standard approaches to fast convolution computation is to use GeMM-based convolution algorithms relying on efficient general matrix multiplication (GeMM) from optimized BLAS libraries. However, commonly used GeMM-based algorithms may cause significant memory overhead or avoid it only at the cost of worse performance. In this paper, we propose a novel convolution algorithm, p-im2col, based on a well-known im2col algorithm that avoids memory overhead by splitting a single multiplication of a large matrix into several multiplications of smaller matrices. We theoretically and experimentally compare our algorithm with two other GeMM-based algorithms: im2col, which is widely used as a baseline, and the memory-efficient kn2row-aa. We measure the inference time of these algorithms on central processing units of x86, x86_64, ARM, and MIPS architectures for a large set of convolutional parameters. The proposed algorithm demonstrates a speedup over im2col and kn2row-aa in a number of cases and a significant reduction in additional memory requirements compared to im2col. Based on our experiments, we present a new convolution algorithm selection scheme that considers memory restrictions, CPU architecture, and convolutional parameters and provides a noticeable advantage over each particular algorithm.