This way the model doesn't have to actively remember everything that was in the input because it will get another look at it later. You can see this in the logit lens, where relative to a vanilla model the model delays the actual prediction.

2 likes 1 replies