Containerized execution of UDFs
Authors:
Open Access
- 1 July 2022
- journal article
- conference paper
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 15 (11) , 3158-3171
Abstract
User-defined functions (UDFs) have long been used as the de facto way to extend the capabilities of data management systems. However, they are restricted to the specificities of each DBMS, and recent demands for advanced analytics have increased the need for complex UDFs that may require execution of arbitrary computation written in any programming language, management of library dependencies, portability across environments and engines, and resource isolation. These requirements go beyond what traditional UDFs were designed for, and have given rise to containerized UDFs that enable encapsulation and portability. However, this approach is nascent and can result in significant performance penalties and usability issues. In this paper, we present the first study that spans all stages of containerized UDFs' life cycle, performance bottlenecks in their execution, and extensibility to support different engines. Our experiments show that the performance of containerized UDF execution can be greatly affected by system design choices and that there are many trade-offs to consider. For example, regarding the method of communication with the containerized UDF, we show that binary-based implementations minimize overheads and are more than 2.4x faster than widely used text-based ones. Adopting a newer general-purpose communication method such as Arrow Flight can improve performance dramatically, causing a minimal ~10% slowdown compared to non-containerized UDFs. Additionally, containerized UDF start times vary wildly due to program size and complexity, from .07s to 7s in our experiments. Our insights can help DBMS developers make appropriate choices based on individual use cases when designing their systems.Keywords
This publication has 22 references indexed in Scilit:
- Cold Start Influencing Factors in Function as a Service2018
- Serving Deep Learning Models in a Serverless Platform2018
- Serverless Computing: An Investigation of Factors Influencing Microservice Performance2018
- Processing Java UDFs in a C++ environment2017
- FID: A Faster Image Distribution System for Docker Platform2017
- Optimization of Complex Dataflows with User-Defined FunctionsACM Computing Surveys, 2017
- An architecture for compiling UDF-centric workflowsProceedings of the VLDB Endowment, 2015
- Optimizing database-backed applications with query synthesis2013
- Opening the black boxes in data flow optimizationProceedings of the VLDB Endowment, 2012
- LAPACK Users' Guide1999