nic 🔹 @nmk.wtf · Jun 17 I also tried every other combination of wiring. It turns out you really want QK from the residual stream, and V from the input.
2 likes 1 replies
?
Replies
I have now been able to further bring down validation loss by optimizing the number of heads, swapping softmax for softmax1 and normalizing the Q and K matrices.