Sam Harsimony

@harsimony.bsky.social

1276 Followers 699 Following

I write about opportunities in science, space, and policy here: https://splittinginfinity.substack.com/

Benchmark | What it tests | Gemma 4 31B | Claude Opus 4.6 | GPT-5.2 / GPT-5.4 ------------------------|----------------------------------------------------|--------------------|------------------------------------|----------------------------------------- MMLU Pro | Graduate-level knowledge, 10-choice | 85.2% | ~82% | ~89-90% (GPT-5.2) GPQA Diamond | PhD-level science reasoning | 84.3% | 91.3% | 93.2% (GPT-5.2 Pro); ~92.8% (GPT-5.4) LiveCodeBench v6 | Contamination-resistant competitive coding | 80.0% | ~76-80% | Not separately reported AIME 2026 (no tools) | Competition math | 89.2% | 100% (2025, w/ Python tools) | 94.6% (2025, no tools) HLE (no tools) | Expert-level multidisciplinary reasoning | 19.5% | ~36.7% | ~27-28% (GPT-5.2) HLE (with search) | Same, with web access | 26.5% | 53.0% (*) | ~41-44% (GPT-5.4 Pro) Tau2 / t2-bench | Multi-turn tool use (Gemma: avg 3 domains; | 76.9% (avg 3) | 98.2% Telecom (Opus 4.5) | 98.7% Telecom (GPT-5.2) | others: Telecom only -- not directly comparable) | | | BigBench Extra Hard | Hard reasoning tasks | 74.4% | Not reported | Not reported MMMLU | Multilingual knowledge Q&A (14 languages) | 88.4% | ~91% | Not cleanly reported MMMU Pro (vision) | Multimodal reasoning: images + text | 76.9% …
So if Data Is All You Need, what does that mean for AI development? A large and low-error dataset is sufficient to train a model to perform any task. John von Neumann had it right; we can make an AI that can do anything so long as we can specify it. We’ll push forward the only way we know how, with bigger datasets and better benchmarks. We’ll find better methods by GitHub Descent. We’ll beg Scott Alexander for training data. Once a benchmark is saturated we’ll make a new one. The benchmark doesn’t teach valuable skills? A stock trading benchmark! A benchmark can’t capture romantic love? A sexbot benchmark! Benchmark can’t identify AGI? ARC-AGI_final_v5! This why machine learning took off in the first place. The culture of openness, competitive benchmarks, and easy-to-steal ideas created a singularity of self-improvement. There’s no limit to what our hive mind can do. Simulate all of biology from gene sequencing data. Strap a Go Pro to everyone’s head to emulate human behavior. Send robots to the stars and start iterating. All takes is the entire semiconductor industry, half of academia, a solar-industrial revolution, and everyone who ever posted on the internet.
"To switch metaphors, people sometimes describe the positive AI-enabled future as 'luxury space communism.' AI would take care of all of our material needs, leaving us to seek value in our lives through family, friendships, competition, hobbies, humor, art, entertainment, or exploration. The super-AI would give us the freedom to pursue all those things, but would not give us the freedom to harm each other, to curtail each others’ freedoms, or to build a bad AI capable of overthrowing it. The super-AI would be a singleton, a monotheistic God or its emissary on earth. Many people say that something would still be missing from this future. After all, we humans would no longer really be needed for anything—for building or advancing or defending civilization. To put a personal fine point on it, my students and colleagues and I wouldn’t needed any more to discover new scientific truths or to write about them. That would all be the AI’s job. I agree that something would be lost here. But on the other hand, what fraction of us are needed right now for these things? Most humans already derive the meaning in their lives from family and community and enjoying art and music and food and things like that. So maybe the remaining fraction of us should just get over ourselves! On the whole, while this might not be the best future imaginable, I would accept it in a heartbeat given the realistic alternatives on offer. Thanks for listening."
Annual hazard rate: '26-'27: 6.2% '28-'29: 7.7% '30-'34: 2.8%
ROCm is already fairly similar to CUDA (by design!), so porting over simple kernels with okay performance is probably well within the capabilities of Claude Code tbh However, porting over complex CUDA kernels like FlashAttention or whatever crazy stuff OpenAI and Anthropic have cooked up internally, and achieving comparable performance to the CUDA version, is extremely hard. And unless somebody has evidence that Claude is able to do that, I would be very very skeptical.
Subheading: RL scaling is laundering the prestige of pretraining scaling Body: With pretraining, we had this extremely clean and general trend in improvement in loss across multiple orders of magnitude of compute (albeit on a power law, which is as weak as exponential growth is strong). People are trying to launder the presitge of pretraining scaling, which was almost as predictable as a physical law of the universe, to justify bullish projections about RLVR, for which we have no well fit publicly known trend. When intrepid researchers do try to piece together the implications from scarce public datapoints, they get quite bearish results. For example, Toby Ord has a great post where he cleverly connects the dots between different o-series benchmark charts, which suggested “we need something like a 1,000,000x scale-up of total RL compute to give a boost similar to a GPT level”.
This is a good point. I think any explanation for why dating apps are bad should also be able to explain: - why social media is full of ragebait - why streaming services are full of shows that string you along - Why its hard to stop eating junk food once you've started Here's my attempt: humans optimize not only for direct rewards (a life partner, in the case of dating apps), but the promise of rewards (going on a date with someone who might become your life partner). This is because the world is complicated and it's hard to learn from a sparse signal (finding a life partner only happens one time), so you have to seek proxies of life partners too (dates, sex, DM's, etc.). People actually get reward from these proxies, that's how we're wired. Dating apps have figured out how to give you the proxies. So now they can make money by both finding you a life partner *and* setting you up on a bunch of dead-end dates. There's an optimal mixture of the two that maximizes profits, so dating companies do both. Many have found life partners on dating apps. This applies to other hyperstimuli I listed above. They strike a balance between selling direct reward and selling proxies for it. It just turns out that our brains put an uncomfortable amount of weight on the proxies and not the actual reward. For more I recommend this post on why hyperstimuli are designed to be unstatisfying: https://www.sympatheticopposition.com/p/hyperstimuli-are-understimulating
Furthermore, to improve the inference economics, such RL models could be trained using LoRA adapters or a similar technique and served alongside thousands of other models, all catered to specific use cases. This multi-tenant serving approach represents a compelling business opportunity for inference providers. Clients hosting their custom LoRA adapters on a provider's infrastructure face significant switching costs when migrating to competitors, as the adapters are optimized for specific serving configurations and client workflows. RLFT is based on unique and nuanced rewards that are very client-specific; unlike standard supervised fine-tuning (SFT), it much much more challenging to replicate it just via in-context learning, making it an even more compelling case for inference providers. We expect the inference markets to further specialize in regard to offered throughput, latency, and pricing. It is only natural for providers of super-fast tokens like Groq and Cerebras to command a much higher premium for the tokens they deliver at few-second latencies and for other providers like NeoCloud specializing in high-latency, high-throughput inference scenarios focused on synthetic data generation. We hope to elaborate on this space in the future text.
quote: "Purely Logical Debate is difficult and annoying. It doesn’t scale. It only works on the subset of people who are willing to talk to you in good faith and smart enough to understand the issues involved. And even then, it only works glacially slowly, and you win only partial victories. What’s the point? Logical debate has one advantage over narrative, rhetoric, and violence: it’s an asymmetric weapon. That is, it’s a weapon which is stronger in the hands of the good guys than in the hands of the bad guys. In ideal conditions (which may or may not ever happen in real life) – the kind of conditions where everyone is charitable and intelligent and wise – the good guys will be able to present stronger evidence, cite more experts, and invoke more compelling moral principles. The whole point of logic is that, when done right, it can only prove things that are true. ..."
A quote from a slate star codex post: "Robert Frost says “A liberal is a man too broad-minded to take his own side in a quarrel”. Ha ha ha. And yet, outside of Saudi Arabia you’ll have a hard time finding a country that doesn’t at least pay lip service to liberal ideas. Stranger still, many of those then go on to actually implement them, either voluntarily or after succumbing to strange pressures they don’t understand. In particular, the history of the past few hundred years in the United States has been a history of decreasing censorship and increasing tolerance. Contra the Reactionaries, feminism isn’t an exception to that, it’s a casualty of it. 1970s feminists were saying that all women need to rise up and smash the patriarchy, possibly with literal smashing-implements. 2010s feminists are saying that if some women want to be housewives, that’s great and their own choice because in a liberal society everyone should be free to pursue their own self-actualization. And that has corresponded to spectacular successes of the specific causes liberals like to push, like feminism, civil rights, gay marriage, et cetera, et cetera, et cetera. A liberal is a man too broad-minded to take his own side in a quarrel. And yet when liberals enter quarrels, they always win. Isn’t that interesting?"