Qiang Fu
Research Subjects
Qiang Fu has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Pavel Velikhov
Institute for System Programming of the Russian Academy of Sciences
4 shared publications - Sergey Dmitrievich Kuznetsov
Институт системного программирования им. В.П. Иванникова РАН, Московский государственный университет им. М.В. Ломоносова, Московский физико-технический институт, НИУ «Высшая школа экономики
4 shared publications - Artem Mikhaylov
366033 Skolkovo Institute of Science and Technology , 121205 Moscow , Russia
1 shared publication - Evgeny Burnaev
Skolkovo Institute of Science and Technology
1 shared publication - Ilya Trofimov
Skolkovo Institute of Science and Technology
1 shared publication - Mikhail Salnikov
Artificial Intelligence Research Institute (AIRI), Moscow, Russia
1 shared publication
Latest Publications
- 1 February 2023
- Published by Pleiades Publishing Ltd in Programming and Computer Software
- Vol. 49 (1) , 1-25
Real-Time Analytics: Benefits, Limitations, and Tradeoffs
Real-time analytics is a relatively new branch of analytics. A common definition of real-time analytics is that it consists in analyzing data as quickly as possible over the most recent data possible. This defines the essence of the fundamental needs of users, but in no way is a specific requirement for the corresponding software systems due to the vagueness of the definition. As a result, different manufacturers of analytical data-management systems and researchers classify real-time analytics systems as extremely different systems, which differ in architecture, functionality, and even timing. The purpose of this article is to analyze the different approaches to providing real-time analytics, their advantages and disadvantages, and the tradeoffs that both designers and users of the systems inevitably have to make.
- 1 February 2023
- Published by Pleiades Publishing Ltd in Programming and Computer Software
- Vol. 49 (1) , 1-25
Real-Time Analytics: Benefits, Limitations, and Tradeoffs
Real-time analytics is a relatively new branch of analytics. A common definition of real-time analytics is that it consists in analyzing data as quickly as possible over the most recent data possible. This defines the essence of the fundamental needs of users, but in no way is a specific requirement for the corresponding software systems due to the vagueness of the definition. As a result, different manufacturers of analytical data-management systems and researchers classify real-time analytics systems as extremely different systems, which differ in architecture, functionality, and even timing. The purpose of this article is to analyze the different approaches to providing real-time analytics, their advantages and disadvantages, and the tradeoffs that both designers and users of the systems inevitably have to make.
- 1 January 2023
- Published by The Russian Academy of Sciences in Программирование
- No. 1,p. 3-31
REAL-TIME ANALYTICS: BENEFITS, LIMITATIONS, AND TRADEOFFS
Real-time analytics is a relatively new branch of analytics. A common definition of real-time analytics is that it consists in analyzing data as quickly as possible over the most recent data possible. This defines the essence of the fundamental needs of users, but in no way is a specific requirement for the corresponding software systems due to the vagueness of the definition. As a result, different manufacturers of analytical datamanagement systems and researchers classify real-time analytics systems as extremely different systems, which differ in architecture, functionality, and even timing. The purpose of this article is to analyze the different approaches to providing real-time analytics, their advantages and disadvantages, and the tradeoffs that both system designers and their users inevitably have to make.
- 1 January 2023
- Published by The Russian Academy of Sciences in Программирование
- No. 1,p. 3-31
REAL-TIME ANALYTICS: BENEFITS, LIMITATIONS, AND TRADEOFFS
Real-time analytics is a relatively new branch of analytics. A common definition of real-time analytics is that it consists in analyzing data as quickly as possible over the most recent data possible. This defines the essence of the fundamental needs of users, but in no way is a specific requirement for the corresponding software systems due to the vagueness of the definition. As a result, different manufacturers of analytical datamanagement systems and researchers classify real-time analytics systems as extremely different systems, which differ in architecture, functionality, and even timing. The purpose of this article is to analyze the different approaches to providing real-time analytics, their advantages and disadvantages, and the tradeoffs that both system designers and their users inevitably have to make.
- Open Access
- 13 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 75205-75218
Learned Query Optimizers: Evaluation and Improvement
Query Optimization is considered to be one of the most important challenges in database management. Existing built-in query optimizers are very complex and rely on various approximations and hand-picked rules. The rise of deep learning and deep reinforcement learning has aided many scientific and industrial fields, providing an opportunity to develop a learnable query optimizer. In this paper, we analyse and improve the state-of-the-art learned query optimizer, Neo for the JOB benchmark on two database systems: PostgreSQL and Huawei GaussDB.We describe our methods, based on combination of Neo, Tree-Transformers, auxiliary tasks, reward weighting. Combinations of these methods improve latency of the found query execution plans. We also conduct a thorough analysis of the resulting execution plans and devise a set of decision-based rules to indicate the cases when the learned optimizer will outperform the built-in one. We also provide a source code for the proposed methods and experiments. Finally, we provide possible directions for further improvement in this field.
- 13 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 75205-75218
Open AccessLearned Query Optimizers: Evaluation and Improvement
Query Optimization is considered to be one of the most important challenges in database management. Existing built-in query optimizers are very complex and rely on various approximations and hand-picked rules. The rise of deep learning and deep reinforcement learning has aided many scientific and industrial fields, providing an opportunity to develop a learnable query optimizer. In this paper, we analyse and improve the state-of-the-art learned query optimizer, Neo for the JOB benchmark on two database systems: PostgreSQL and Huawei GaussDB.We describe our methods, based on combination of Neo, Tree-Transformers, auxiliary tasks, reward weighting. Combinations of these methods improve latency of the found query execution plans. We also conduct a thorough analysis of the resulting execution plans and devise a set of decision-based rules to indicate the cases when the learned optimizer will outperform the built-in one. We also provide a source code for the proposed methods and experiments. Finally, we provide possible directions for further improvement in this field.
- Open Access
- 1 January 2021
- Published by Institute for System Programming of the Russian Academy of Sciences in Proceedings of the Institute for System Programming of the RAS
- Vol. 33 (3) , 171-198
Real-Time Analytics, Hybrid Transactional/Analytical Processing, In-Memory Data Management, and Non-Volatile Memory
Аналитика в реальном времени, гибридная транзакционная/аналитическая обработка, управление данными в основной памяти и энергонезависимая память
- 1 January 2021
- Published by Institute for System Programming of the Russian Academy of Sciences in Proceedings of the Institute for System Programming of the RAS
- Vol. 33 (3) , 171-198
Open AccessReal-Time Analytics, Hybrid Transactional/Analytical Processing, In-Memory Data Management, and Non-Volatile Memory
Аналитика в реальном времени, гибридная транзакционная/аналитическая обработка, управление данными в основной памяти и энергонезависимая память
- 1 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 78-90
Real-time analytics, hybrid transactional/analytical processing, in-memory data management, and non-volatile memory
These days, real-time analytics is one of the most often used notions in the world of databases. Broadly, this term means very fast analytics over very fresh data. Usually the term comes together with other popular terms, hybrid transactional/analytical processing (HTAP) and in-memory data processing. The reason is that the simplest way to provide fresh operational data for analysis is to combine in one system both transactional and analytical processing. The most effective way to provide fast transactional and analytical processing is to store an entire database in memory. So on the one hand, these three terms are related but on the other hand, each of them has its own right to life. In this paper, we provide an overview of several in-memory data management systems that are not HTAP systems. Some of them are purely transactional, some are purely analytical, and some support real-time analytics. Then we overview nine in-memory HTAP DBMSs, some of which don't support real-time analytics. Existing real-time in-memory HTAP DBMSs have very diverse and interesting architectures although they use a number of common approaches: multiversion concurrency control, multicore parallelization, advanced query optimization, just in time compilation, etc. Additionally, we are interested whether these systems use non-volatile memory, and, if yes, in what manner. We conclude that an emergence of new generation of NVM will greatly stimulate its use in in-memory HTAP systems.
- 1 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 78-90
Real-time analytics, hybrid transactional/analytical processing, in-memory data management, and non-volatile memory
These days, real-time analytics is one of the most often used notions in the world of databases. Broadly, this term means very fast analytics over very fresh data. Usually the term comes together with other popular terms, hybrid transactional/analytical processing (HTAP) and in-memory data processing. The reason is that the simplest way to provide fresh operational data for analysis is to combine in one system both transactional and analytical processing. The most effective way to provide fast transactional and analytical processing is to store an entire database in memory. So on the one hand, these three terms are related but on the other hand, each of them has its own right to life. In this paper, we provide an overview of several in-memory data management systems that are not HTAP systems. Some of them are purely transactional, some are purely analytical, and some support real-time analytics. Then we overview nine in-memory HTAP DBMSs, some of which don't support real-time analytics. Existing real-time in-memory HTAP DBMSs have very diverse and interesting architectures although they use a number of common approaches: multiversion concurrency control, multicore parallelization, advanced query optimization, just in time compilation, etc. Additionally, we are interested whether these systems use non-volatile memory, and, if yes, in what manner. We conclude that an emergence of new generation of NVM will greatly stimulate its use in in-memory HTAP systems.