Johannes Weiss

@jweiss.io

681 Followers 514 Following

Computer programmer (http://github.com/weissi). Distributed systems, ML infrastructure/eval, Swift, previously #SwiftNIO — ♥ snowboarding London, UK

Anthropic $33B capex, $6.5B revenue
Wholewheat bread labelled 'German bread'
In every region of the globe, Solar Turbines is a world leader in mid-range industrial gas turbines for use in power generation, natural gas compression, and pumping systems. Our systems last for decades, and so do our service and support.
Lego set parts list next to handwritten inventory checklist.
Sunshine every day for the next 10 days in London
A spreadsheet with 8 columns and 22 rows. The first row contains 1 through 8. The last row contains 2399993 through 2400000. All the other cells are empty.
Cherry Capital Airport's address is 727 Fly Don’t Drive.
Most outrageous value?
Evals really matter # Anthropic’s Amanda Askell (responsible for much of the work behind Claude’s Character): The boring yet crucial secret behind good system prompts is test-driven development. You don’t write down a system prompt and find ways to test it. You write down tests and find a system prompt that passes them. It’s become abundantly clear over the course of 2024 that writing good automated evals for LLM-powered systems is the skill that’s most needed to build useful applications on top of these models. If you have a strong eval suite you can adopt new models faster, iterate better and build more reliable and useful product features than your competition.
A drum l've been banging for a while is that LLMs are power-user tools - they're chainsaws disguised as kitchen knives. They look deceptively simple to use-how hard can it be to type messages to a chatbot? — but in reality you need a huge depth of both understanding and experience to make the most of them and avoid their many pitfalls. If anything, this problem got worse in 2024. We've built computer systems you can talk to in human language, that will answer your questions and usually get them right! ... depending on the question, and how you ask it, and whether it's accurately reflected in the undocumented and secret training set.
An idea that surprisingly seems to have stuck in the public consciousness is that of "model collapse". This was first described in the paper The Curse of Recursion: Training on Generated Data Makes Models Forget in May 2023, and repeated in Nature in July 2024 with the more eye-catching headline Al models collapse when trained on recursively generated data. The idea is seductive: as the internet floods with Al-generated slop the models themselves will degenerate, feeding on their own output in a way that leads to their inevitable demise! That's clearly not happening. Instead, we are seeing Al labs increasingly train on synthetic content - deliberately creating artificial data to help steer their models in the right way.
Is this infrastructure necessary? DeepSeek v3's $6m training cost and the continued crash in LLM prices might hint that it's not. But would you want to be the big tech executive that argued NOT to build out this infrastructure only to be proven wrong in a few years' time?
Vintage tram that's now used to drink mulled wine.
Flighty's Calendar Sync options
Expensive Rimowa suitcase for 12 bottles of wine.
286 computer with MS-DOS 5.0.0