Will Held @williamheld.com · May 11

Last fall, I naively thought that I had a decent scaling recipe built on top of muP and a few other works on hyperparameter transfer! But when we scaled to larger (8B+) scales with compute support from the Google TPU Research Cloud, things quickly went wrong!

3 likes 1 replies

?

Replies

Will Held · May 11

There were two red flags in the runs from that initial sweep: suspiciously large weight norms and learning rates! For the first, I switched to Kaiyue Wen's Hyperball optimizers. For the second, I found the answer in Apple's Complete(d)P paper: arxiv.org/abs/2512.22382