Sophie Hao

@cinnamonlab.ai

265 Followers 330 Following

Assistant professor of Linguistics and Data Science at Boston University. NLP, computational linguistics, interpretability, social bias and fairness. she/her. https://www.notaphonologist.com/

A plot showing the rate of growth of spending on LLM training predicted by our model over the next few years. LLM training spend grows faster when marginal demand diminishes less in terms of LLM quality. The actual rate of investment growth for LLM training is higher than the rate predicted if demand for tokens is logarithmic in LLM quality, and lower than the rate predicted if demand for tokens is linear in LLM quality.
Plots showing how LLM size (n*) and total training expenditure C*_train scale with data efficiency (b) in LLM quality/training token. As data efficiency increases, LLM size always increases, but total training expenditure may decrease.
Plots showing how LLM size (n*) and total training expenditure C*_train scale with parameter efficiency (a) in LLM quality/param. Both quantities decrease as parameter efficiency increases.
Plots showing how LLM size (n*) and total training expenditure C*_train scales with hardware efficiency (E) in FLOPs/$. Numerically, we find that both quantities scale sublinearly with E. Analytically, our asymptotic bounds are at most slightly superlinear.
Mathematical expressions giving asymptotic upper bounds (Big-O notation) on the profit-optimal LLM size (n*) and training data size (d*) based on hardware efficiency (E), parameter efficiency (a), and data efficiency (b). The full expression is available at https://arxiv.org/abs/2605.16430
A screenshot of the title of a research paper. The title is "A Theory of Training Profit-Optimal LLMs," and the authors are Sophie Hao from Boston University and William Merrill from the Allen Institute of AI.