Luca Soldaini 🎀 @soldaini.net · Jan 3 OLMo 2 tech report is out! We get in the weeds with this one, with 50+ pages on 4 crucial components of LLM development pipeline:
47 likes 3 replies
?
Replies
Would love to hear more about those stability mitigations. Even with careful gradient clipping and warmup, we still see occasional NaN explosions at scale - what's your secret sauce?
📌
💪 Stability How do you ensure that your pretrain run doesn't blow up? We spent months perfecting our OLMo recipe---it takes many targeted mitigations and expensive experiments to get it right… well now you don't have to!