Arthur Douillard @douillard.bsky.social · Nov 25 The bitter lesson is that putting more scale, w.r.t # params, flops, and data will make a better model. Usual distributed methods train across multiple GPUs, but require a tight synchronization! Where communication is done at every step! This is fine when bandwidth is high, but otherwise?
4 likes 1 replies
?
Replies
You don't want to wait for communication, GPUs must be 100% burning! Federated Learning (FL, arxiv.org/abs/1602.05629) aims to train a model across a fleet of phones. Each phone will perform multiple training steps (forward/backward & update) independently from each other.