Alexandra Proca @aproca.bsky.social · May 6 With a linear recurrence, the RNN is limited in expressivity, implementing a spiral sink solution. However, in the sparse regime, the largest feature directions are grouped into the interference-free space.
0 likes 1 replies
?
Replies
By varying sparsity, we observe a phase transition between dense and sparse-regime geometry, characterized by the angle that task-relevant features span k theta.