KRKamran Razavi

Kamran Razavi

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Kamran Razavi has a total of 16 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Bridging CPU and GPU Autoscaling for Cost-Efficient Inference Serving

    by Mehran Salmani, Kamran Razavi, Peter Amthor, Boris Koldehofe

    Autoscaling mechanisms have been used by inference serving systems to maintain service quality and efficiently use cloud resources (CPUs/GPUs). Horizontal scaling-based mechanisms have a responsiveness issue due to cold-start delays. In-place CPU vertical scaling adopted by recent works avoids cold-start; however, it is constrained by hardware limits and model parallelizability, and is limited to CPUs, which are efficient for low workloads but cost-inefficient for large workloads. Thus, combining CPUs and GPUs under dynamic workloads is worthwhile. However, enabling responsive switching between them is challenging. We propose VCG, an inference serving system that utilizes different resources based on the workload, and enables a smooth transition from CPUs to GPUs and vertically scaling GPUs (adjusting the GPU percentage) by forking a new process with enough GPU compute resources using CUDA MPS, and delivering inference requests to the child process through inter-process communication pipes. When the capacity of service is (nearly) exhausted, VCG uses horizontal scaling to provide scalability headroom for future steps. We evaluate a prototype of VCG under two different workloads and show that it reduces SLO violations up to 8× compared to a baseline using horizontal scaling of GPU instances under a bursty workload. It can also reduce the cost by around 60% compared to two CPU-based autoscaling baselines under a dynamic (non-bursty) workload.

  • Open Access

    IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency

    by Saeid Ghafouri, Kamran Razavi, Mehran Salmani, Alireza Sanaee, Tania Lorido Botran, Lin Wang, Joseph Doyle, Pooyan Jamshidi


  • Towards Democratic Computing

    by Max Mühlhäuser, Nikolaos Alexopoulos, Uwe Gropengießer, Kamran Razavi, Lin Wang

  • Open Access

    Sponge

    by Kamran Razavi, Saeid Ghafouri, Max Mühlhäuser, Pooyan Jamshidi, Lin Wang

    Mobile and IoT applications increasingly adopt deep learning inference to provide intelligence. Inference requests are typically sent to a cloud infrastructure over a wireless network that is highly variable, leading to the challenge of dynamic Service Level Objectives (SLOs) at the request level. This paper presents Sponge, a novel deep learning inference serving system that maximizes resource efficiency while guaranteeing dynamic SLOs. Sponge achieves its goal by applying in-place vertical scaling, dynamic batching, and request reordering. Specifically, we introduce an Integer Programming formulation to capture the resource allocation problem, providing a mathematical model of the relationship between latency, batch size, and resources. We demonstrate the potential of Sponge through a prototype implementation and preliminary experiments and discuss future works.


  • Open Access

    Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

    by Mehran Salmani, Saeid Ghafouri, Alireza Sanaee, Kamran Razavi, Max Mühlhäuser, Joseph Doyle, Pooyan Jamshidi, Mohsen Sharifi

    The use of machine learning (ML) inference for various applications is growing drastically. ML inference services engage with users directly, requiring fast and accurate responses. Moreover, these services face dynamic workloads of requests, imposing changes in their computing resources. Failing to right-size computing resources results in either latency service level objectives (SLOs) violations or wasted computing resources. Adapting to dynamic workloads considering all the pillars of accuracy, latency, and resource cost is challenging. In response to these challenges, we propose InfAdapter, which proactively selects a set of ML model variants with their resource allocations to meet latency SLO while maximizing an objective function composed of accuracy and cost. InfAdapter decreases SLO violation and costs up to 65% and 33%, respectively, compared to a popular industry autoscaler (Kubernetes Vertical Pod Autoscaler).


  • Open Access

    Operator as a Service: Stateful Serverless Complex Event Processing

    by Manisha Luthra, Sebastian Hennig, Kamran Razavi, Lin Wang, Boris Koldehofe

    Complex Event Processing (CEP) is a powerful paradigm for scalable data management that is employed in many real-world scenarios such as detecting credit card fraud in banks. The so-called complex events are expressed using a specification language that is typically implemented and executed on a specific runtime system. While the tight coupling of these two components has been regarded as the key for supporting CEP at high performance, such dependencies pose several inherent challenges as follows. (1) Application development atop a CEP system requires extensive knowledge of how the runtime system operates, which is typically highly complex in nature. (2) The specification language dependence requires the need of domain experts and further restricts and steepens the learning curve for application developers. In this paper, we propose CEPLESS, a scalable data management system that decouples the specification from the runtime system by building on the principles of serverless computing. CEPLESS provides operator as a service and offers flexibility by enabling the development of CEP application in any specification language while abstracting away the complexity of the CEP runtime system. As part of CEPLESS, we designed and evaluated novel mechanisms for in-memory processing and batching that enables the stateful processing of CEP operators even under high rates of ingested events. Our evaluation demonstrates that CEPLESS can be easily integrated into existing CEP systems like Apache Flink while attaining similar throughput under a high scale of events (up to 100K events per second) and dynamic operator update in up to 238 ms.