David Heineman @davidheineman.com · Aug 19 (3/6) 🔎 We landed on a simple metric - the signal-to-noise ratio - the ratio between the dispersion of scores from models, and the variation of final checkpoints of a single model. This allows estimating SNR with a small number of models (around 50 models) at any compute scale!
1 likes 1 replies
?
Replies
(4/6) 🧐 How do we know SNR is meaningful? We can (1) calculate % of models ranked correctly at small vs. 1B scale and (2) fit scaling laws to predict the task performance. SNR is predictive of better decision accuracy, and tasks with lower noise have lower scaling law error!