Eugene Yan

@eugeneyan.com

10796 Followers 331 Following

RecSys, AI, Engineering; Principal Applied Scientist @ Amazon. Led ML @ Alibaba, Lazada, Healthtech Series A. Writing @ eugeneyan.com, aiteratelabs.com.

✅ selfcheckgpt.jpg: 226.15KB → 60.53KB (73.24% reduction) ✅ query-processing.jpg: 95.74KB → 42.84KB (55.25% reduction) ✅ sldc-specialists.jpg: 30.88KB → 12.37KB (59.95% reduction) ✅ feature-store-ad.png: 157.24KB → 57.73KB (63.29% reduction) ✅ llm-patterns-aieng-2023-v0-004.jpg: 132.81KB → 43.50KB (67.24% reduction) ✅ google-user-intent.png: 46.67KB → 24.80KB (46.86% reduction) ✅ quy-nguyen.jpeg: 4.27KB → 1.98KB (53.61% reduction) ✅ fbi-tab2.jpg: 396.10KB → 128.28KB (67.61% reduction) ✅ ey-fastball.png: 4.94KB → 0.46KB (90.71% reduction) ✅ favicon-16x16.png: 0.60KB → 0.26KB (56.96% reduction) ✅ android-chrome-192x192.png: 11.50KB → 2.96KB (74.30% reduction) ✅ apple-touch-icon.png: 10.15KB → 2.55KB (74.88% reduction) ✅ android-chrome-512x512.png: 33.22KB → 6.72KB (79.77% reduction) ✅ favicon-32x32.png: 1.35KB → 0.45KB (66.45% reduction) ✅ 404-8.jpg: 43.90KB → 21.81KB (50.32% reduction) ✅ 404-9.jpg: 42.80KB → 16.86KB (60.60% reduction) ✅ 404.jpg: 25.67KB → 6.37KB (75.20% reduction) ✅ 404-11.jpg: 91.48KB → 13.71KB (85.02% reduction) ✅ 404-10.jpg: 309.60KB → 34.35KB (88.91% reduction) ✅ 404-12.jpg: 68.38KB → 50.75KB (25.79% reduction) ✅ 404-13.jpg: 73.79KB → 38.56KB (47.74% reduction) ✅ 404-14.jpg: 86.34KB → 45.91KB (46.82% reduction) ✅ 404-1.jpg: 25.67KB → 6.37KB (75.20% reduction) ✅ 404-2.jpg: 113.98KB → 67.45KB (40.82% reduction) ✅ 404-3.jpg: 38.06KB → 13.68KB (64.04% reduction) ✅ 404-7.jpg: 55.02KB → 26.54KB (51.77% reduction) ✅ 404-6.jpg: 45.98KB → 15.45KB (66.41% reduction) ✅ 404-4.jpg: 94.63KB → 50.33KB (46.82% reduction) ✅ 404-5.jpg: 48.97KB → 17.69KB (63.87% reduction) ================================================== SUMMARY STATISTICS ================================================== Total files converted: 1002 Total original size: 122.78MB Total WebP size: 38.75MB Total size reduction: 84.03MB (68.44%) Average size reduction per file: 68.44% Proportional savings: 122.78MB → 38.75MB ==================================================
Image of recommender widget at the bottom of posts on eugeneyan.com
bsky share button
Effective Evals for AI products
The Art of Doing Science and Engineering: Learning to Learn by Richard Hamming
airbnb generate-validate loop
4.3 Neural Benchmarks on Stories The design of the NarrativeQA dataset makes the straight-forward application of the existing neural architectures computationally infeasible, as this would require running an recurrent neural network on sequences of hundreds of thousands of time steps or computing a distribution over the entire input for attention, as is common. We split the task into two steps: first, we retrieve a small number of relevant passages from the story using an IR system, and subsequently, apply one of
If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on? 1. Two metrics that come to mind • Faithfulness: Grounding of answers in document's content. Not to be confused with correctness—an answer can be correct (based on updated information) but not faithful to the document. Sub-metric: Precision of citations • Helpfulness: Usefulness (directly addresses the question with enough detail and explanation) and completeness (does not omit important details); an answer can be faithful but not helpful if too brief or doesn't answer the question • Evaluate separately: Faithfulness = binary label -> LLM-evaluator; Helpfulness = pairwise comparisons -> reward model
Two-stage recommender system
To Eugene, My favorite AI writer Chip Huyen
Any resources you’d recommend on the topic of writing? Writing, Briefly Write Like You Talk Write Simply Why Everyone Should Write Writing Better Easy Reading Is Damn Hard Writing Mise en Place Writing Amazon Writing Style Tips Some Blogging Myths Some Tactics for Writing in Public Some Thoughts on Writing 10 years of professional blogging – what I’ve learned Lessons from content marketing myself (aka blogging) for five years Make Your Writing Work Harder For You What I learned writing a book How Jeff Bezos Turned Narrative into Amazon’s Competitive Advantage Seemingly Paradoxical Rules of Writing What I Did Not Learn About Writing In School What I Learned from Writing Online - For Fellow Non-Writers How to Write Better with The Why, What, How Framework How to Write Design Docs for Machine Learning Systems Writing Tools: 55 Essential Strategies for Every Writer
My reaction is that there is an evaluation crisis. I don't really know what metrics to look at right now. MMLU was a good and useful for a few years but that's long over. SWE-Bench Verified (real, practical, verified problems) I really like and is great but itself too narrow. Chatbot Arena received so much focus (partly my fault?) that LLM labs have started to really overfit to it, via a combination of prompt mining (from API requests), private evals bombardment, and, worse, explicit use of rankings as training supervision. I think it's still ~ok and there's a lack of "better", but it feels on decline in signal. There's a number of private evals popping up, an ensemble of which might be one promising path forward. In absence of great comprehensive evals I tried to turn to vibe checks instead, but I now fear they are misleading and there is too much opportunity for confirmation bias, too low sample size, etc., it's just not great. TLDR my reaction is I don't really know how good these models are right now.
https://cdn.openai.com/gpt-4-5-system-card.pdf
slide from anthropic talk
👋 Hi there! I work on <redacted> at Airbnb. I really enjoy your writing. As a personal project I started trying to write something kind of like AI reading club. I wanted a more powerful version of the "x ray" feature in Kindle, because sometimes when I pick up a new book in a series or return to a book after a break, I cannot remember all of the characters or the plot. I work mainly on <redacted> and not so much app development, and I really struggled with how to build my AI x ray. AI reading club is awesome and I was curious about how you built the retrieval pipeline, any preprocessing you did to the text, etc. Thanks for publishing so much great work. My team and many of the machine learning engineers whom I support at Airbnb frequently share your posts. Best regards
Books on AI Reading Club and the number of messages on them.
"...I joined a hedged fund, Renaissance Technologies, I'll make a comment about that. It's funny that I think the most important thing to do on data analysis is to do the simple things right. So, here's a kind of non-secret about what we did at renaissance: in my opinion, our most important statistical tool was simple regression with one target and one independent variable. It's the simplest statistical model you can imagine. Any reasonably smart high school student could do it. Now we have some of the smartest people around, working in our hedge fund, we have string theorists we recruited from Harvard, and they're doing simple regression. Is this stupid and pointless? Should we be hiring stupider people and paying them less? And the answer is no. And the reason is nobody tells you what the variables you should be regressing [are]. What's the target. Should you do a nonlinear transform before you regress? What's the source? Should you clean your data? Do you notice when your results are obviously rubbish? And so on. And the smarter you are the less likely you are to make a stupid mistake. And that's why I think you often need smart people who appear to be doing something technically very easy, but actually usually not so easy.] [[at] my hedge fund, which was not a very big company, we had 7 Phd's just cleaning data and organizing the databases]"
Screenshot from google analytics showing 60 active users per minute
Finally, if we need help with a term or character that was previously mentioned, Dewey can help with a summary of the term so we don’t have to look it up ourselves.
If you've stopped reading a book for a while, it can be challenging to pick it up again and remember what you've read. To help with this, it can help with summarizing the book up to the current page and refresh our memory, highlighting major themes, characters, and concepts.
It can also help with creating quizzes / flashcards. The goal here is to test our knowledge and improve retention.
With the context, it can answer simple queries via "Explain" and "Discuss". The goal is to keep us in flow while reading, instead of having to reread other sections of the book or open a web browser for our queries.
It understands our context either via the text we select or the page we're on. This explicit context is displayed during discussions. At the same time, behind the scenes, it can also retrieve and consider the rest of the book as implicit context.
deciding between "actually taking some time off" and "work on personal projects and call it relaxing"
2hr 17min average daily screen time
comments that talk about the inevitability of ai failing
To validate this, I visited a sample of the critique accounts and found at least a dozen profiles of writers, artists, musicians, and other creatives, some of whom added explicit anti-AI declarations to their bios.
At first glance, the primary issue seemed to be the lack of consent in data collection. Some comments expressed anger at having their data “scraped” or “stolen” without permission. (Note: The accusation is incorrect—there was no scraping or stealing involved. Posts on Bluesky are public and the Bluesky firehose API is open for all to consume.)
Reactions against the release of one-million-bluesky-posts
Day 1 of hitting the slopes this winter 🏂
Academic benchmarks are not your tasks.
Old quant types (some base model types require these): - Q4_0: small, very high quality loss - legacy, prefer using Q3_K_M - Q4_1: small, substantial quality loss - legacy, prefer using Q3_K_L - Q5_0: medium, balanced quality - legacy, prefer using Q4_K_M - Q5_1: medium, low quality loss - legacy, prefer using Q5_K_M New quant types (recommended): - Q2_K: smallest, extreme quality loss - not recommended - Q3_K: alias for Q3_K_M - Q3_K_S: very small, very high quality loss - Q3_K_M: very small, very high quality loss - Q3_K_L: small, substantial quality loss - Q4_K: alias for Q4_K_M - Q4_K_S: small, significant quality loss - Q4_K_M: medium, balanced quality - recommended - Q5_K: alias for Q5_K_M - Q5_K_S: large, low quality loss - recommended - Q5_K_M: large, very low quality loss - recommended - Q6_K: very large, extremely low quality loss - Q8_0: very large, extremely low quality loss - not recommended - F16: extremely large, virtually no quality loss - not recommended - F32: absolutely huge, lossless - not recommended
Database of resources for applying ml
UI for the LLMOps Database with the search query of "eval"
The key is authenticity. Don’t do this just for marketing—do it because you’re genuinely interested in learning from others and building on their ideas. It’s not hard to find things to be excited about. I’m amazed by how few people take this approach. It’s both effective and fun.
A video demonstrating how Rotary Positional Embedding work
Nassim Taleb on 'True Wealth' Vishal Khandelwal, safalniveshak.com Nassim Taleb is one of my favourite authors, and his Antifragile is one of my favourite books. One of this book's chapters that interests me particularly is titled 'Via Negativa'. Here, Taleb argues that the solution to many problems in life is by removing things, not adding things. For example, here is a list of things Taleb counts as constituents of true wealth that are all about subtracting things (via negativa) from life than adding - Worriless sleeping Clear conscience Reciprocal gratitude Absence of envy Good appetite Muscle strength Physical energy Frequent laughs No meals alone No gym classes Some physical labor Good bowel movements No meeting rooms Periodic surprises I could check twelve from this list (let the ones I didn't check remain a secret). What about you? What in the list remains getting checked for you?
Writeups on machine learning systems, techniques, ml & engineering, and weekend prototypes
Read through this making flashcards as you to: https://eugeneyan.com/writing/llm-patterns/ Then spin up a RAG-enhanced chatbot using pgvector on your favourite subject, and keep improving it when you learn about cool techniques --- Lots of people can get impressive demos up and running, but if you want to run AI products in production, you're going to have to do system evals. System evals make sure your product is doing what it says on the box with unquantifiable qualities. We wrote a zine on system evals without jargon: https://forestfriends.tech Eugene Yan has written extensively on it https://eugeneyan.com/writing/evals/ Hamel has as well. https://hamel.dev/blog/posts/evals/
This is a taxonomy for task space that I find useful when thinking about what we need to do to solve alignment. Crisp tasks are more reasoning/system-2 based. Whether a response is good is typically precisely defined. Reasonable, knowledgeable people don’t disagree about it. Examples: math, coding competitions, physics questions, logic puzzles, basic factual knowledge, … Fuzzy tasks are more intuition/system-1 based. Whether a response is good is typically somewhat vaguely defined or could fall within a range. Reasonable, knowledgeable people may disagree about it. Examples: distinguishing cats and dogs in pictures, recognizing strong go moves, poetry writing, assistant helpfulness ratings, … In contrast to fuzzy logic, where this name is borrowed from, here crisp and fuzzy don’t refer to the output space: preferences comparisons have a small number of discrete allowed values, and image classification typically uses a discrete space. Crisp tasks like coding competitions have a output space that involves hundreds of discrete tokens, and this is the same output space of poetry writing, a fuzzy task. There are many crisp tasks that we can evaluate very reliably, and thus it’s feasible to train on them extensively. In contrast, fuzzy tasks are often particularly difficult to evaluate reliably, and we currently don’t know how to do this at a superhuman level.
Northern lights, streak of green with an igloo
tools for brainstorming and development such as obsidian, zotero, and cursor
brew install chruby-fish ruby-install ruby-build brew install rbenv # workaround for fish shell set --universal fish_user_paths $fish_user_paths ~/.rbenv/shims rbenv global 3.3.5 rbenv rehash
View from plane on flight to alaska
Potentially nitpicky but competitive advantage in AI goes not so much to those with data but those with a data engine: iterated data aquisition, re-training, evaluation, deployment, telemetry. And whoever can spin it fastest. Slide from Tesla to ~illustrate but concept is general
It’s hard to understand now, the Atari RL paper of 2013 and its extensions was the by far dominant meme. One single general learning algorithm discovered an optimal strategy to Breakout and so many other games. You just had to improve and scale it enough. My recollection of the memetics is that Yann LeCun was one prominent person who really didn’t care much and talked about the cake over and over again, where RL was just the final cherry on top with representation learning as the meat and supervised learning the icing, and he was conceptually exactly right about that at least with today’s stack and hindsight (pretraining = meat, SFT = icing, RLHF = cherry, ie the basic ChatGPT training pipeline). Which is fun because today he really doesn’t care much for LLMs either. (But for reasons that I tbh don’t always fully follow.)
Yann LeCun’s analogy of intelligence being a cake of self-supervised, supervised, and RL
Closing the feedback loop
evals are surprisingly often all you need
user profile with invalid handle
summarized message from apple intelligence
Hi Eugene. I saw your blog on LLM-As-A-Judge a couple months ago and it's become my goto for sending to customers when they want to learn more. Had no idea you were at Amazon until today. Just wanted to thank you for the content. Blog for reference: https://eugeneyan.com/writing/llm-evaluators/ Have a great weekend!
7. Evals are a differentiator and moat. Over the past two years, teams with solid evals have been able to continuously ship reliable, delightful experiences. No one regrets investing in a robust evaluation framework.
Optimization metrics for LLM-evaluators
Dropdown menu with claude-3-haiku, claude-3.5-haiku, and gpt-4o-mini.
Behind the scenes: How AlignEval was built As a frontend development newbie (this is my first TypeScript app), I relied on the wisdom of the crowd, polling Twitter and LinkedIn for advice. I also built the same app five times to get a feel for several frameworks. Specifically, I tried FastHTML, Next.js, SvelteKit, and FastAPI + HTML (my go-to for prototyping). From early prototypes, Next.js seemed to be a good fit. Intuitive, scalable, and the perfect excuse to learn TypeScript. And thanks to Cursor, my beginner questions were answered easily: What’s the difference between const and let? What is a prop? It also made it easier to build UX components and fix bugs 2-3x faster, keeping velocity and motivation high.