nic 🔹 @nmk.wtf · Jun 17 Over the past week I trained a bunch of picoGPTs (~50M params) on single GPUs trying to get minimal loss with a fixed number of epochs. The biggest architectural impact I found was replacing some of the transformer Value tensors with Value tensors that read from the original input.
10 likes 2 replies
?
Replies
This way the model doesn't have to actively remember everything that was in the input because it will get another look at it later. You can see this in the logit lens, where relative to a vanilla model the model delays the actual prediction.
How does this work? Like a separate skip connection through V tensors or ?