| title | FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth |
|---|---|
| authors | Josef Chen, Erim Hayretci |
| date | 2026-08-20 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.20574 |
| upvotes | 2 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and family (14,418 model-task cells total), eliminating differential missingness from the leaderboard. The FlavourBench Score is the equal-family mean of the frozen task scores. We use 50,000 anchor-cluster bootstrap replicates for simultaneous 95% score bands and 100,000 sign-flip draws for all 351 paired model contrasts, with Holm control. The two independently compiled panels correlate at r = 0.89 (rank rho = 0.80). Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0-69.2); 101 of 351 model pairs are resolved. The release includes the prompts, all portfolio score maps, raw responses, exact routes, content hashes, and an offline verifier that reconstructs every result. |
cite
@misc{chen2026flavourbench,
title={FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth},
author={Josef Chen and Erim Hayretci},
year={2026},
eprint={2608.20574},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.20574}
}