Abstract
The same machine learning model can be served by GPUs and AI accelerators that differ by more than an order of magnitude in throughput, price, memory capacity, and power, yet the best execution configuration depends jointly on the model, its tail-latency service-level objective (SLO), its memory footprint, and the load it must sustain. This paper presents XAccel, a constraint-guided framework that selects and adapts inference configurations across heterogeneous accelerators under latency, cost, and energy budgets. XAccel (i) builds a unified, SLO-aware performance descriptor from 2,173 audited MLPerf Inference runs (v4.0-v6.0) covering 28 accelerator types and 34 workloadscenario pairs, including p95/p99 latency, time-to-first-token and time-per-output-token percentiles, and wall power reconstructed for 88 runs; (ii) selects configurations by solving an SLO-, memory-, and budget-constrained cost-energy program over measured operating points; (iii) transfers performance to unmeasured accelerators and models through a datasheetinformed elasticity model with hierarchical anchor calibration; and (iv) re-selects fleets under week-long production traces of Azure LLM services. Across 30 workload-SLO settings, choosing the fastest, the cheapest-per-hour, or the incumbent H100 accelerator raises the cost per inference by 56-152% on average relative to XAccel. Zero-shot transfer predicts SLO-bounded throughput with 28% mean absolute percentage error and preserves 94% of selection decisions at 1.6% mean cost regret, and the cost-optimal accelerator for a new model is found after 2.2 measurements instead of 4.5. Under real production load, XAccel lowers weekly cost by 69-82% and energy by 69-80% relative to a peak-provisioned H100 fleet for LLM services, with at most 0.51% SLO-risk minutes.