| title | GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving |
|---|---|
| authors | Alessio Ricci Toniolo, Rome Thorstenson, Abinaya Dinesh |
| date | 2026-06-30 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2602.11688 |
| upvotes | 5 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO. |
cite
@misc{toniolo2026gorgo,
title={GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving},
author={Alessio Ricci Toniolo and Rome Thorstenson and Abinaya Dinesh},
year={2026},
eprint={2602.11688},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2602.11688}
}