titleChoosing Quantized Code LLMs by File Size and Energy per Solved Task
authorsAditi Patodiya
date2026-11-13
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.17605
upvotes2 (Hugging Face Daily Papers, 2026-11-21)
linksabs pdf hf
cite
abstractA developer running a code assistant on a laptop downloads one model file, and local inference moves the energy of every request onto that machine. A common rule says a larger model at lower precision tends to beat a smaller one at higher precision. We ran 24 GGUF files of Qwen2.5-Coder at four sizes on a fanless MacBook Air M2 and priced each in energy per solved task on 542 EvalPlus tasks. The result is a selection guide indexed by Ollama's listed file size, with an evidence label on each pick. At 1 and 2 GB it picks the q4_K_M files of the 1.5B and 3B models, both on the correctness and energy frontier under every accounting. At 3 GB as listed, the larger-model rule picks 7B-q2_K, the only measured 7B file that fits, and there it does not pay. That file brings no significant gain in solves over the 2 GB pick, 3B-q4_K_M, and the 90% CI limits any 3B-q4_K_M deficit to 2.4 points. Yet 7B-q2_K needs 2.03 times the energy per solved task on CPU, GPU and memory rails, or 1.66 to 1.81 times on whole-system power. On HumanEval+, five alternating session pairs give a ratio of 1.925 against 1.956 in the main grid. At 8 GB, 7B-q6_K buys a significant gain over the 2 GB pick at a higher energy price. From 1.5B up, q4_K_M trails the highest precision measured at its size by less than 3 points (90% CI). We release per-task results, power logs and the analysis script.

bibtex

@misc{patodiya2026choosing,
  title={Choosing Quantized Code LLMs by File Size and Energy per Solved Task},
  author={Aditi Patodiya},
  year={2026},
  eprint={2611.17605},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.17605}
}