TRTheodoros Rekatsinas

Theodoros Rekatsinas

*This is an algorithmically generated author record based on publicly available information.

Theodoros Rekatsinas has a total of 424 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Principal Component Networks: Utilizing Low-Rank Activation Structure to Reduce Parameters Early in Training

    by Roger Waleffe, Theodoros Rekatsinas

    Recent works show that overparameterized neural networks contain small subnetworks that exhibit comparable accuracy to the full model when trained in isolation. These results highlight the potential to reduce the computational costs of deep neural network training without sacrificing generalization performance. Initial approaches for finding these small networks relied on expensive multi-round train-and-prune procedures, limiting their practical potential, but more recent work identifies subnetworks using structured pruning techniques early in training. In this article, we study network activations, rather than network weights, and find that hidden layer activations in overparameterized networks exist primarily in subspaces smaller than the actual model width. We further notice that these subspaces can be identified early in training. Based on these observations, we show how to efficiently find small networks that exhibit similar accuracy to their overparameterized counterparts after only a few training epochs. We term these network architectures Principal Component Networks (PCNs). PCNs compress individual layers by retaining only the high variance linear combinations of channels—defined by the principal components of the layer inputs—a key difference from structured pruning techniques which focus on individual channel pruning using localized channel measurements. We evaluate PCNs on CIFAR-10 and ImageNet for VGG and ResNet style architectures and compare against existing methods for subnetwork identification during early training. We find that PCNs consistently reduce parameter counts with little accuracy loss, thus providing the potential to reduce the computational costs of deep neural network training. Beyond model compression, we also connect our observation regarding hidden layer activations to the feature representations learned by neural networks and discuss areas for future work.


  • Open Access

    Record Fusion via Inference and Data Augmentation

    by Alireza Heidari, George Michalopoulos, Ihab F. Ilyas, Theodoros Rekatsinas

    We introduce a learning framework for the problem of unifying conflicting data in multiple records referring to the same entity—we call this problem “record fusion.” Record fusion generalizes two known problems: “data fusion” and “golden record.” Our approach expresses record fusion as a learning problem over probabilistic models. In contrast to prior approaches, our method achieves high performance with or without the records source information and outperforms state-of-the-art baselines. Furthermore, we show how our learned fusion model can solve the problem of scarcity of training data. On multiple datasets, we show that our framework fuses records with an average precision of ∼98% when source information is available and ∼94% without source information across a diverse array of datasets. We compare our approach to a comprehensive collection of data fusion and entity consolidation methods, ranging from source information–related methods to approaches that do not need any source information. We show that our approach can achieve an average improvement of ∼20/∼45 precision points with/without source information. Our data augmentation method improves previous approaches an average of ∼10 precision points.


  • Open Access

    Co-design Hardware and Algorithm for Vector Search

    by Wenqi Jiang, Shigang Li, Yu Zhu, Johannes De Fine Licht, Zhenhao He, Runbin Shi, Cedric Renggli, Shuai Zhang, Theodoros Rekatsinas, Torsten Hoefler, et al.

    Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousands of queries per second on petabyte-scale document datasets by evaluating vector similarities between encoded query texts and web documents. As performance demands for vector search systems surge, accelerated hardware offers a promising solution in the post-Moore's Law era. We introduce FANNS, an end-to-end and scalable vector search framework on FPGAs. Given a user-provided recall requirement on a dataset and a hardware resource budget, FANNS automatically co-designs hardware and algorithm, subsequently generating the corresponding accelerator. The framework also supports scale-out by incorporating a hardware TCP/IP stack in the accelerator. FANNS attains up to 23.0× and 37.2× speedup compared to FPGA and CPU baselines, respectively, and demonstrates superior scalability to GPUs, achieving 5.5× and 7.6× speedup in median and 95th percentile (P95) latency within an eight-accelerator configuration. The remarkable performance of FANNS lays a robust groundwork for future FPGA integration in data centers and AI supercomputers.


  • Open Access

    EnergAt: Fine-Grained Energy Attribution for Multi-Tenancy

    by Hongyu Hè, Michal Friedman, Theodoros Rekatsinas

    In the post-Moore's Law era, relying solely on hardware advancements for automatic performance gains is no longer feasible without increased energy consumption, due to the end of Dennard scaling. Consequently, computing accounts for an increasing amount of global energy usage, contradicting the objective of sustainable computing. The lack of hardware support and the absence of a standardized, software-centric method for the precise tracing of energy provenance exacerbates the issue. Aiming to overcome this challenge, we argue that fine-grained software energy attribution is attainable, even with limited hardware support. To support our position, we present a thread-level, NUMA-aware energy attribution method for CPU and DRAM in multi-tenant environments. The evaluation of our prototype implementation, EnergAt, demonstrates the validity, effectiveness, and robustness of our theoretical model, even in the presence of the noisy-neighbor effect. We envisage a sustainable cloud environment and emphasize the importance of collective efforts to improve software energy efficiency.


  • Top Cited Papers
    Open Access

    High-Throughput Vector Similarity Search in Knowledge Graphs

    by Jason Mohoney, Anil Pacaci, Shihabur Rahman Chowdhury, Ali Mousavi, Ihab F. Ilyas, Umar Farooq Minhas, Jeffrey Pound, Theodoros Rekatsinas

    There is an increasing adoption of machine learning for encoding data into vectors to serve online recommendation and search use cases. As a result, recent data management systems propose augmenting query processing with online vector similarity search. In this work, we explore vector similarity search in the context of Knowledge Graphs (KGs). Motivated by the tasks of finding related KG queries and entities for past KG query workloads, we focus on hybrid vector similarity search (hybrid queries for short) where part of the query corresponds to vector similarity search and part of the query corresponds to predicates over relational attributes associated with the underlying data vectors. For example, given past KG queries for a song entity, we want to construct new queries for new song entities whose vector representations are close to the vector representation of the entity in the past KG query. But entities in a KG also have non-vector attributes such as a song associated with an artist, a genre, and a release date. Therefore, suggested entities must also satisfy query predicates over non-vector attributes beyond a vector-based similarity predicate. While these tasks are central to KGs, our contributions are generally applicable to hybrid queries. In contrast to prior works that optimize online queries, we focus on enabling efficient batch processing of past hybrid query workloads. We present our system, HQI, for high-throughput batch processing of hybrid queries. We introduce a workload-aware vector data partitioning scheme to tailor the vector index layout to the given workload and describe a multi-query optimization technique to reduce the overhead of vector similarity computations. We evaluate our methods on industrial workloads and demonstrate that HQI yields a 31× improvement in throughput for finding related KG queries compared to existing hybrid query processing approaches.


  • Open Access

    Growing and Serving Large Open-domain Knowledge Graphs

    by Ihab F. Ilyas, Jp Lacerda, Yunyao Li, Umar Farooq Minhas, Ali Mousavi, Jeffrey Pound, Theodoros Rekatsinas, Chiraag Sumanth

    Applications of large open-domain knowledge graphs (KGs) to real-world problems pose many unique challenges. In this paper, we present extensions to Saga our platform for continuous construction and serving of knowledge at scale. In particular, we describe a pipeline for training knowledge graph embeddings that powers key capabilities such as fact ranking, fact verification, a related entities service, and support for entity linking. We then describe how our platform, including graph embeddings, can be leveraged to create a Semantic Annotation service that links unstructured Web documents to entities in our KG. Semantic annotation of the Web effectively expands our knowledge graph with edges to open-domain Web content which can be used in various search and ranking problems. Finally, we leverage annotated Web documents to drive Open-domain Knowledge Extraction. This targeted extraction framework identifies important coverage issues in the KG, then finds relevant data sources for target entities on the Web and extracts missing information to enrich the KG. Finally, we describe adaptations to our knowledge platform needed to construct and serve private personal knowledge on-device. This includes private incremental KG construction, cross- device knowledge sync, and global knowledge enrichment.


  • Top Cited Papers
    Open Access

    MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks

    by Roger Waleffe, Jason Mohoney, Theodoros Rekatsinas, Shivaram Venkataraman

    We study training of Graph Neural Networks (GNNs) for large-scale graphs. We revisit the premise of using distributed training for billion-scale graphs and show that for graphs that fit in main memory or the SSD of a single machine, out-of-core pipelined training with a single GPU can outperform state-of-the-art (SoTA) multi-GPU solutions. We introduce MariusGNN, the first system that utilizes the entire storage hierarchy---including disk---for GNN training. MariusGNN introduces a series of data organization and algorithmic contributions that 1) minimize the end-to-end time required for training and 2) ensure that models learned with disk-based training exhibit accuracy similar to those fully trained in memory. We evaluate MariusGNN against SoTA systems for learning GNN models and find that single-GPU training in MariusGNN achieves the same level of accuracy up to 8× faster than multi-GPU training in these systems, thus, introducing an order of magnitude monetary cost reduction. MariusGNN is open-sourced at www.marius-project.org.


  • Open Access

    Machine Learning and Data Cleaning: Which Serves the Other?

    by Ihab F. Ilyas, Theodoros Rekatsinas

    The last few years witnessed significant advances in building automated or semi-automated data quality, data cleaning and data integration systems powered by machine learning (ML). In parallel, large deployment of ML systems in business, science, environment and various other areas started to realize the strong dependency on the quality of the input data to these ML models to get reliable predictions or insights. That dual relationship between ML and data cleaning has been addressed by many recent research works under terms such as “Data cleaning for ML” and “ML for automating data cleaning and data preparation”. In this article, we highlight this symbiotic relationship between ML and data cleaning and discuss few challenges that require collaborative efforts of multiple research communities.


  • Open Access

    Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inference

    by Zifan Liu, Zhechun Zhou, Theodoros Rekatsinas

    Data corruption is an impediment to modern machine learning deployments. Corrupted data can severely bias the learned model and can also lead to invalid inferences. We present, Picket, a simple framework to safeguard against data corruptions during both training and deployment of machine learning models over tabular data. For the training stage, Picket identifies and removes corrupted data points from the training data to avoid obtaining a biased model. For the deployment stage, Picket flags, in an online manner, corrupted query points to a trained machine learning model that due to noise will result in incorrect predictions. To detect corrupted data, Picket uses a self-supervised deep learning model for mixed-type tabular data, which we call PicketNet. To minimize the burden of deployment, learning a PicketNet model does not require any human-labeled data. Picket is designed as a plugin that can increase the robustness of any machine learning pipeline. We evaluate Picket on a diverse array of real-world data considering different corruption models that include systematic and adversarial noise during both training and testing. We show that Picket consistently safeguards against corrupted data during both training and deployment of various models ranging from SVMs to neural networks, beating a diverse array of competing methods that span from data quality validation models to robust outlier-detection models.