Anand Iyer
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Anand Iyer has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
- Haiying Shen
Department of Computer science, University of Virginia, Charlottesville, Virginia, USA
2 shared publications - Ravi Netravali
Princeton University, Princeton, USA
2 shared publications - Rui Pan
Princeton University, Princeton, USA
2 shared publications - Yinwei Dai
Princeton University, Princeton, New Jersey, UNITED STATES
2 shared publications - Ganesh Ananthanarayanan
Microsoft, Redmond, USA
1 shared publication - Ikhyun An
Georgia Institute of Technology, Atlanta, Georgia, USA
1 shared publication
Latest Publications
- Open Access
- 28 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 18-35
S andhi : Fine-Grained Merging for Memory Efficient Multi-Model Serving
- 28 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 18-35
Open AccessS andhi : Fine-Grained Merging for Memory Efficient Multi-Model Serving
- Open Access
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 507-523
Flex: Fast, Accurate DNN Inference on Low-Cost Edges Using Heterogeneous Accelerator Execution
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 507-523
Open AccessFlex: Fast, Accurate DNN Inference on Low-Cost Edges Using Heterogeneous Accelerator Execution
- Open Access
- 25 November 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Networking
- Vol. 2 (CoNEXT4) , 1-18
PACER: Accelerating Distributed GNN Training Using Communication-Efficient Partition Refinement and Caching
Despite recent breakthroughs in distributed Graph Neural Network (GNN) training, large-scale graphs still generate significant network communication overhead, decreasing time and resource efficiency. Although recently proposed partitioning or caching methods try to reduce communication inefficiencies and overheads, they are not sufficiently effective due to their sampling pattern-agnostic nature. This paper proposes a Pipelined Partition Aware Caching and Communication Efficient Refinement System (Pacer), a communication-efficient distributed GNN training system. First, Pacer intelligently estimates each partition's access frequency to each vertex by jointly considering the sampling method and graph topology. Then, it uses the estimated access frequency to refine partitions and caching vertices in its two-level cache (CPU and GPU) to minimize data transfer latency. Furthermore, Pacer incorporates a pipeline-based minibatching method to mask the effect of the network communication. Experimental results on real-world graphs show that Pacer outperforms state-of-the-art distributed GNN training system in training time by 40% on average.
- 25 November 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Networking
- Vol. 2 (CoNEXT4) , 1-18
Open AccessPACER: Accelerating Distributed GNN Training Using Communication-Efficient Partition Refinement and Caching
Despite recent breakthroughs in distributed Graph Neural Network (GNN) training, large-scale graphs still generate significant network communication overhead, decreasing time and resource efficiency. Although recently proposed partitioning or caching methods try to reduce communication inefficiencies and overheads, they are not sufficiently effective due to their sampling pattern-agnostic nature. This paper proposes a Pipelined Partition Aware Caching and Communication Efficient Refinement System (Pacer), a communication-efficient distributed GNN training system. First, Pacer intelligently estimates each partition's access frequency to each vertex by jointly considering the sampling method and graph topology. Then, it uses the estimated access frequency to refine partitions and caching vertices in its two-level cache (CPU and GPU) to minimize data transfer latency. Furthermore, Pacer incorporates a pipeline-based minibatching method to mask the effect of the network communication. Experimental results on real-world graphs show that Pacer outperforms state-of-the-art distributed GNN training system in training time by 40% on average.
- Open Access
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 624-639
Improving DNN Inference Throughput Using Practical, Per-Input Compute Adaptation
Machine learning inference platforms continue to face high request rates and strict latency constraints. Existing solutions largely focus on compressing models to substantially lower compute costs (and time) with mild accuracy degradations. This paper explores an alternate (but complementary) technique that trades off accuracy and resource costs on a perinput granularity: early exit models, which selectively allow certain inputs to exit a model from an intermediate layer. Though intuitive, early exits face fundamental deployment challenges, largely owing to the effects that exiting inputs have on batch size (and resource utilization) throughout model execution. We present E3, the first system that makes early exit models practical for realistic inference deployments. Our key insight is to split and replicate blocks of layers in models in a manner that maintains a constant batch size throughout execution, all the while accounting for resource requirements and communication overheads. Evaluations with NLP and vision models show that E3 can deliver up to 1.74× improvement in goodput (for a fixed cost) or 1.78× reduction in cost (for a fixed goodput). Additionally, E3's goodput wins generalize to autoregressive LLMs (2.8--3.8×) and compressed models (1.67×).
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 624-639
Open AccessImproving DNN Inference Throughput Using Practical, Per-Input Compute Adaptation
Machine learning inference platforms continue to face high request rates and strict latency constraints. Existing solutions largely focus on compressing models to substantially lower compute costs (and time) with mild accuracy degradations. This paper explores an alternate (but complementary) technique that trades off accuracy and resource costs on a perinput granularity: early exit models, which selectively allow certain inputs to exit a model from an intermediate layer. Though intuitive, early exits face fundamental deployment challenges, largely owing to the effects that exiting inputs have on batch size (and resource utilization) throughout model execution. We present E3, the first system that makes early exit models practical for realistic inference deployments. Our key insight is to split and replicate blocks of layers in models in a manner that maintains a constant batch size throughout execution, all the while accounting for resource requirements and communication overheads. Evaluations with NLP and vision models show that E3 can deliver up to 1.74× improvement in goodput (for a fixed cost) or 1.78× reduction in cost (for a fixed goodput). Additionally, E3's goodput wins generalize to autoregressive LLMs (2.8--3.8×) and compressed models (1.67×).
- Open Access
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 607-623
Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5--91.5% and 10.0--24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6--77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 607-623
Open AccessApparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5--91.5% and 10.0--24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6--77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.