Researchers identifies "Super Weights" which are at most a handful of weights (amongst billions!). Models end up generating complete gibberish if these weights are removed, underlining the brittleness of LLMs and importance and careful consideration of outlier features. arxiv.org/pdf/2411.07191

17 likes 2 replies