Jeremy Howard @howard.fm · Dec 19 Seven months ago, @bclavie.bsky.social kicked things off, and soon Benjamin Warner & @nohtow.bsky.social joined him as project co-leads. I don't think anyone quite knew what we were getting in to… It turns out that training a new, SoTA model from scratch is actually pretty hard. Who knew? 🤷
26 likes 1 replies
?
Replies
We created ModernBERT because encoder-only architectures are ideal for the kinds of real-world problems that come up every day, like retrieval (e.g. for RAG), classification (such as content moderation), & entity extraction. But they're under-appreciated, with little investment.