Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving

TY - GEN

T1 - Apparate

T2 - 30th ACM Symposium on Operating Systems Principles, SOSP 2024

AU - Dai, Yinwei

AU - Pan, Rui

AU - Iyer, Anand

AU - Li, Kai

AU - Netravali, Ravi

N1 - Publisher Copyright: © 2024 Copyright held by the owner/author(s).

PY - 2024/11/15

Y1 - 2024/11/15

N2 - Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5 - 91.5% and 10.0 - 24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6 - 77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.

AB - Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5 - 91.5% and 10.0 - 24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6 - 77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.

UR - http://www.scopus.com/inward/record.url?scp=85214165737&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85214165737&partnerID=8YFLogxK

U2 - 10.1145/3694715.3695963

DO - 10.1145/3694715.3695963

M3 - Conference contribution

T3 - SOSP 2024 - Proceedings of the 2024 ACM SIGOPS 30th Symposium on Operating Systems Principles

SP - 607

EP - 623

BT - SOSP 2024 - Proceedings of the 2024 ACM SIGOPS 30th Symposium on Operating Systems Principles

PB - Association for Computing Machinery, Inc

Y2 - 4 November 2024 through 6 November 2024

ER -