Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 194-199
Abstract
Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which extends the hardware scheduling of a simultaneously multithreaded (SMT) core to provide an "express-lane" execution context for short tasks, protecting them from queuing behind rare, long ones. As tasks reach predefined service cutoffs, ESMT preempts and migrates them to the subsequent queue to be serviced by the next SMT execution lane, thereby preventing Head-of-Line (HoL) blocking. We further propose an enhanced variant of ESMT that allows execution lanes to work-steal from each other to maximize utilization. Our evaluation shows that ESMT with work stealing reduces tail latency over a conventional SMT core by an average of 56% and 67% under moderate (40%) and high (70%) system loads, respectively.Keywords
This publication has 26 references indexed in Scilit:
- Profiling a warehouse-scale computer2015
- Few-to-ManyACM SIGPLAN Notices, 2015
- Where does the time go? characterizing tail latency in memcached2015
- Using RDMA efficiently for key-value servicesACM SIGCOMM Computer Communication Review, 2014
- The tail at scaleCommunications of the ACM, 2013
- Scale-out processorsACM SIGARCH Computer Architecture News, 2012
- The gem5 simulatorACM SIGARCH Computer Architecture News, 2011
- On Choosing a Task Assignment Policy for a Distributed Server SystemJournal of Parallel and Distributed Computing, 1999
- Task assignment in a distributed system (extended abstract)ACM SIGMETRICS Performance Evaluation Review, 1998
- Exploiting choiceACM SIGARCH Computer Architecture News, 1996