Pınar Tözün

Traditionally, deploying database systems required acquiring and managing large, expensive servers on-premise. These servers feature a memory and storage hierarchy, including CPU caches, DRAM memory, HDDs and/or SSDs, and sometimes tape. For almost 40 years, Gray and Putzolu’s five-minute rule [4, 10, 12, 13] has helped guide system architects to the break-even point between memory caching and direct local storage access for these systems. The rise of cloud computing has significantly changed the way systems are deployed. Cloud services offer on-demand computing and storage, removing the need for companies to acquire and manage their own hardware. Furthermore, the cloud has introduced a new layer of the storage hierarchy for database systems: object stores . This durability and low cost have made object stores ubiquitous in modern cloud-native databases and data infrastructures [5, 6, 16]. We believe a rule of thumb similar to the five-minute rule is needed for object caches and storage for disaggregated cloud data system designs. However, it is not straightforward to adapt the established rules to the cloud as they presume fixed hardware, while, in the cloud, resources are dynamic and costs are determined by usage. One can argue that accessing object stores over the network incurs higher latency and lower bandwidth than directly attached storage devices, placing them at the bottom of the storage hierarchy. However, advances in cloud storage and networking have significantly impacted cost-effectiveness, latency variability, and dynamic workload optimization, challenging these traditional assumptions. For example, in non-storage-optimized AWS instances, network bandwidth often exceeds local storage read bandwidth, and writing to local storage devices tends to be half as fast as reading, while writing to S3 can match network bandwidth. This paper is a summary of our earlier work work [9] that proposes a cost model and new rules of thumb to help system designers determine when caches become cost-effective for analytical workloads in the cloud. While perhaps unsurprising, our analysis on AWS shows that caches are beneficial when a system makes (1) two requests per hour for latency-sensitive workloads, or (2) seven requests per second for non-latency-sensitive workloads. These results are consistent with and help explain the near ubiquity of object store caches in cloud analytics systems.

Collocating deep learning training tasks improves GPU utilization but risks resource contention, severe slowdowns, and out-of-memory (OOM) failures. Accurate memory estimation is essential for robust collocation, and GPU utilization estimation — a key proxy for contention — enables interference-aware scheduling. Existing GPU memory estimators span three paradigms -analytical models, CPU-side libraries, and ML-based estimators - each with distinct limitations: dependence on detailed model specifications, intrusive integration, poor generalization, and varying latency overhead. GPU heterogeneity further complicates estimation, as identical tasks can exhibit different memory footprints across hardware generations. GPU utilization remains comparatively understudied, further complicated by the non-additive utilization metrics and GPU heterogeneity. We conduct a systematic analysis of representative memory estimators from each paradigm - Horus [22], PyTorch FakeTensor [3], and our lightweight ML-based estimator - evaluating accuracy, generalizability, and overhead. We construct a synthetic dataset spanning MLPs, CNNs, and Transformers with controlled architectural variations, and train MLP- and Transformer-based estimators for memory prediction, and experiment with utilization estimation. Our evaluation reveals key tradeoffs and validates estimators against real-world unseen models. Significant challenges remain: analytical models lack generalization and cannot easily be extended to new GPU architectures or accurately reflect memory optimization savings; CPU-side libraries impose intrusive integration overhead; and both analytical and ML-based estimators rely on model specifications or computation graphs, limiting generalization across diverse model architectures and GPU hardware variants. We release all datasets, tools, and artifacts to support further research.

Training deep learning models is a repetitive and resource-intensive process. Data scientists often train several models before landing on a set of parameters (e.g., hyper-parameter tuning) and model architecture (e.g., neural architecture search), among other things that yield the highest accuracy. The computational efficiency of these training tasks depends highly on how well the training data is supplied to the training process. The repetitive nature of these tasks results in the same data processing pipelines running over and over, exacerbating the need for and costs of computational resources. In this paper, we present T ensor S ocket to reduce the computational needs of deep learning training by enabling simultaneous training processes to share the same data loader. T ensor S ocket mitigates CPU-side bottlenecks in cases where the collocated training workloads have high throughput on GPU, but are held back by lower data-loading throughput on CPU. T ensor S ocket achieves this by reducing redundant computations and data duplication across collocated training processes and leveraging modern GPU-GPU interconnects. While doing so, T ensor S ocket is able to train and balance differently-sized models and serve multiple batch sizes simultaneously and is hardware- and pipeline-agnostic in nature. Our evaluation shows that T ensor S ocket enables scenarios that are infeasible without data sharing, increases training throughput by up to 100%, and when utilizing cloud instances, achieves cost savings of 50% by reducing the hardware resource needs on the CPU side. Furthermore, T ensor S ocket outperforms the state-of-the-art solutions for shared data loading such as CoorDL and Joader; it is easier to deploy and maintain and either achieves higher or matches their throughput while requiring fewer CPU resources.

In real-world machine learning (ML) pipelines, datasets are continuously growing. Models must incorporate this new training data to improve generalization and adapt to potential distribution shifts. The cost of model retraining is proportional to how frequently the model is retrained and how much data it is trained on, which makes the naive approach of retraining from scratch each time impractical. We present Modyn, a data-centric end-to-end machine learning platform. Modyn's ML pipeline abstraction enables users to declaratively describe policies for continuously training a model on a growing dataset. Modyn pipelines allow users to apply data selection policies (to reduce the number of data points) and triggering policies (to reduce the number of trainings). Modyn executes and orchestrates these continuous ML training pipelines. The system is open-source and comes with an ecosystem of benchmark datasets, models, and tooling. We formally discuss how to measure the performance of ML pipelines by introducing the concept of composite models, enabling fair comparison of pipelines with different data selection and triggering policies. We empirically analyze how various data selection and triggering policies impact model accuracy, and also show that Modyn enables high throughput training with sample-level data selection.

Satellites have become more widely available due to the reduction in size and cost of their components. As a result, there has been an advent of smaller organizations having the ability to deploy satellites with a variety of data-intensive applications to run on them. One popular application is image analysis to detect, for example, land, ice, clouds, etc. for Earth observation. However, the resource-constrained nature of the devices deployed in satellites creates additional challenges for this resource-intensive application. In this paper, we present our work and lessons-learned on building an Image Processing Unit (IPU) for this satellite. We first highlight the resource constraints based on a deployed satellite performing machine learning on satellite imagery in orbit, including the required latency, power budget, and the network bandwidth limitations driving the need for such a solution. We then investigate the performance of a variety of edge devices (comparing CPU, GPU, TPU, and VPU) for deep-learning-based image processing on satellites. Our goal is to identify devices that are flexible when the workload changes while satisfying the power and latency constraints of satellites. Our results demonstrate that hardware accelerators such as ASICs and GPUs are essential for meeting the latency requirements. However, state-of-the-art edge devices with GPUs may draw too much power for deployment on a satellite.

Standardized benchmarks are crucial to ensure a fair comparison of performance across systems. While extremely valuable, these benchmarks all use a setup where the workload is well-defined and known in advance. Unfortunately, this has led to overly-tuning data management systems for particular benchmark workloads such as TPC-H or TPC-C. As a result, benchmarking results frequently do not reflect the behavior of these systems in many real-world settings since workloads often significantly vary from the "known" benchmarking workloads. To address this issue, we present surprise benchmarking, a complementary approach to the current standardized benchmarking where "unknown" queries are exercised during the evaluation.