Simon P. Couch

@simonpcouch.com

3861 Followers 210 Following

he/him - writing statistical software at Posit, PBC (née RStudio)🥑 simonpcouch.com, @simonpcouch elsewhere

A bar plot showing scores for several frontier models. The two leaders, Gemini 3.5 Flash and Claude Fable 5, score in the mid-high teens. Gemini 3.6 Flash is around 10%. Models from OpenAI cluster at the bottom, never eclipsing 10%.
A dumbbell plot, one row per model, comparing accuracy on artifact plots the model drew with a geom_smooth() overlay versus without. For nearly every model the 'with overlay' point sits well to the left of the 'without' point; Claude Fable 5 falls from about a quarter correct to zero, and Gemini 3.5 Flash from about a quarter to under a tenth.
Blog post header reading 'RStudio Beta 2 (v0.93),' posted April 11, 2011 in News by jjallaire with 32 comments. The opening text announces that RStudio Beta 2 is available for download and reflects feedback from the R community, with a link to the release notes for full details.
Two screencasts of Posit Assistant, labeled "Before" and "With /auto mode". In "Before", the user is asked to manually Allow many commands. In "With /auto mode", the commands are determined to be safe by a classifier model and approved automatically.
An opening slide with title "It’s (still) very bad to be wrong: Agents for correct, transparent, and reproducible data analysis." To the right of the title is a series of AI-related R package hex stickers.
In between tool calls, Fable 5 says "ponds and bridges are good; energy's imputed line is invisible in that noisy cloud — tightening it".
A horizontal bar chart comparing AI models' performance on the bluffbench eval. The chart shows percentages of correct (blue) and incorrect (orange) answers when interpreting counterintuitive data visualizations. The best score across all models is Fable 5 (Medium thinking), around 72%. The second best scores is Gemini 3.5 Flash (High thinking), around 60%. Without thinking enabled, most models cluster around 25%.
A horizontal bar chart comparing AI models' performance on the bluffbench eval. The chart shows percentages of correct (blue) and incorrect (orange) answers when interpreting counterintuitive data visualizations. The best score across all models is from Gemini 3 Flash, around 65%. Opus 4.8 with thinking set to high is around 55%, and without thinking is closer to 25%, a few percentage points over its predecessors.
I ask "Can I add an MCP server just in this project?" In the response, the agent reads the skill file and then tells me briefly how I could do so.
A screencap of the data cleaning mode feature. After I say "Let's clean the data", the agent begins running a bunch of checks to surface data quality issues. Once it has a few to surface, it presents me with multiple-choice options describing the issue and proposing remedies.
Horizontal bar chart comparing agentic coding reliability across three groups. Frontier models (Claude Sonnet 4.5, Gemini Pro 3.1, GPT 4.1) score 80-100% correct. Four-months-ago's local models (Qwen 3 14B, GPT OSS 20B, Mistral 3.1 24B) all score 0%. Today's local models (Gemma 4 26B-A4B, Qwen 3.5 35B-A3B) both åscore 90%.
Horizontal bar chart comparing agentic coding reliability across three groups. Frontier models (Claude Sonnet 4.5, Gemini Pro 3.1, GPT 4.1) score 80-100% correct. Four-months-ago's local models (Qwen 3 14B, GPT OSS 20B, Mistral 3.1 24B) all score 0%. Today's local models (Gemma 4 26B-A4B, Qwen 3.5 35B-A3B) both score 90%.
Model graded performance on an autocomplete task. Latency measures roundtrip of ~2,500 input tokens and ~250 output tokens. Gemma 4 26B scores second only to Claude Haiku 4.5, though at a substantially higher latency than our Qwen3-8B deployments. Our Qwen3-8B deployments hover around 150ms, while Gemma 4 26B is closer to 750ms and frontier labs fastest models around 1250ms.
Three replies to the quoted post, all from AI bot accounts. They each respond with a couple generic, edgy-but-affirmative sentences.
A scatter plot of median latency in milliseconds on the x-axis against mean model-graded quality score on the y-axis, with frontier lab models colored by provider: OpenAI in light blue, Anthropic in burnt orange, and Google in green. The new GPT 5.4 Mini and Nano entries (bold labels) sit at roughly 1,100–1,300 ms latency with scores of 4.1 and 3.75 respectively, notably higher than their GPT 4.1 Nano and GPT 5 Nano predecessors which score around 3.5 at similar latencies. Claude Haiku 4.5 achieves the highest score (~4.5) at the highest latency (~1,500 ms), while Gemini 3.1 Flash-Lite falls between the OpenAI clusters in both dimensions; the non-frontier models appear in grey in the background.
A screencast of RStudio. In a "Posit Assistant" pane in RStudio with a chat UI, I ask a model to take a look at some data in an R package. Then, I ask it to add a plot it's made to the package README.
In a web chat UI, I type "Tell me about RStudio." Seemingly immediately after responding, a couple pages of text is visible, almost as if it had been copied and pasted.
A screencast of an RStudio session. I'm pressing a key command, briefly describing what I want, waiting for a response to stream in, and then running the code and looking at the result. The streamed response is highlighted, so I can easily undo it or kick of the same process again, allowing me to easily iterate on a piece of code.
Scrolling through the post over a section that explains a plot. On the x-axis is the response latency, and on the y-axis is the quality of the response. Our system is in the top left, being the lowest latency and highest quality system that we identified through the evaluation.
A screenshot of an RStudio window. On the left-hand side is a new pain called Posit Assistant. The Posit Assistant had recently run code making a lat-lon plot of Washington state, colored by whether the point had been marked as forested or not.
A pink and blue graphic reading "apply for our opportunity scholarship to posit::conf(2026)."
A code editor containing R code with a panel on the right showing a comment from 'Tidy reviewer'. On the left, the workspace setup section loads various libraries including tidyverse, extraDistr, MASS, cmdstanr, and bayesplot. Three lines (tidyr, purrr, and ggplot2) are highlighted in red, indicating they've been flagged. The 'Tidy Reviewer' panel displays feedback explaining that these three packages are redundant because tidyverse already includes them, suggesting their removal to simplify dependencies.
A bar chart showing the electricity use of several daily activities with the subtitle "The 'typical query' is not a useful way to think about coding agents' energy use." The bar for a 'typical ChatGPT query' is not even visible. My median Claude Code session is somewhere between the average US household per minute and toasting bread for three minutes. My median day with Claude Code is something like running a dishwasher.
A screencast of the RStudio IDE with a new "Posit Assistant" pane. Response to a request for an analysis of mtcars, the agent makes a few plots and then places the code inside of a file.
A plot titled "Agentic coding reliability" with subtitle "When choosing models to power coding tools, you get what you pay for." The x axis reads "Percent Correct." Notably, all three "Local" models (24B or less) score 0 on the eval.
A brown border collie lying in the snow in the foreground. In the background, a small sledding hill with all sorts of folks enjoying it.
A bar chart showing the electricity use of several daily activities with the subtitle "The 'typical query' is not a useful way to think about coding agents' energy use." The bar for a 'typical ChatGPT query' is not even visible. My median Claude Code session is somewhere between the average US household per minute and toasting bread for three minutes. My median day with Claude Code is something like running a dishwasher.
A screenshot showing an RStudio window. Notable, a "Posit Assistant" pane on the right side shows an AI-powered chat interface using Claude Sonnet 4.5 that has generated ggplot2 code for a "Happiness Trends Over Time" visualization, with a rendered line chart preview displayed below the code.
A hexagonal logo for "infer" featuring a green Douglas Fir tree centered on a mountain landscape background with blue-gray peaks and white snow-capped ridges. The word "infer" appears in a modern, lowercase black font at the bottom of the hexagon, which has a dark blue border.
A scatter plot comparing AI models by R coding accuracy (percent correct) versus total cost in USD, with points colored by provider (Anthropic in purple, Google in yellow, OpenAI in blue). Claude Opus 4.5 and GPT-5.2 Pro achieve the highest accuracy (~70%), but Claude Opus 4.5 does so at roughly one-sixth the cost of GPT-5.2 Pro.
A screencast of an RStudio session. An .R file is open in the editor with a function definition. The user selects various subsets of the function and, after selecting from a dropdown menu, the helper assistant converts erroring code and drafts function documentation. The responses take a second or two to complete.
A diagram containing four boxes with arrows linking each pointing left to right. The boxes read, in order, "R interface," "driver manager," "ODBC driver," and "DBMS." The left-most box, R interface, contains three smaller components, labeled "dbplyr," "DBI," and "odbc."
Scatter plot revealing the cost-performance tradeoff across nine LLMs. Claude models occupy the upper-middle, with perfect accuracy and moderate cost. Gemini models tend to be cheaper but less performant. OpenAI models are more variable. The local (and budget Flash and Mini) models cluster in the lower-left, with minimal cost but zero performance. The models closest to the 'top left', being cheap but powerful, are Claude Haiku 4.5 and Gemini Pro 2.5.
A screenshot of the beginning of a conversation with databot, where I say 'Read blog/2025-12-03-wrapped-databot/Library.xml and turn it into a tibble' and the agent begins working.
A screencast of vitals' Log Viewer, an app that allows users to drill into the details of LLM evaluations. The example evaluation shows an image of a Christmas village being described by an LLM.
An RStudio screencast. I run: ``` library(ellmer) gemini_3 <- chat_google_gemini( model = "gemini-3-pro-preview", api_args = list( generationConfig = list(thinkingConfig = list(thinkingLevel = "low")) ) ) side::kick(gemini_3) ``` This launches the sidekick app in the RStudio sidebar, with Gemini powering the app. Then, I ask "Load the forested data from the forested package and make 3 plots." Gemini then makes a series of tool calls that executes the request relatively cleanly.
A stacked bar chart illustrates model hallucination on built-in R datasets like mtcars, showing the proportion of correct versus incorrect responses from three language models: GPT-5, Gemini Pro 2.5, and Claude Sonnet 4.5. All three models demonstrate similarly poor performance, with approximately 90-95% incorrect responses (shown in coral/orange) and only 5-10% correct responses (shown in blue), indicating a tendency to report expected characteristics rather than accurately describing the plotted data.
A screencast of an RStudio session. When I first run `side::kick()`, I see: Which provider/model would you like to use with `side::kick`? 1: Anthropic (Claude Sonnet 4.5) 2: OpenAI (GPT 4.1) 3: Google Gemini (Gemini 2.5 Pro) 4: GitHub (GPT 4.1) 5: Some other provider/model Once I make a selection, I'm then prompted: Store this configuration... 1: Just for this R session 2: In this and future R sessions
A screencast of an RStudio session. A script tool-fetch_skill.R is open in the editor. A call to `side::kick()` in the console launches a non-blocking shiny chat app in a full-height viewer pane. The user then pastes a snippet of code into the chat dialog and says 'refactor this into a helper'. After calling multiple tools to locate the code, the chat app proposes two file edits in a git diff-based UI that refactor the code as desired.
A title slide reading "Practical AI for data science" and my name and title.
A before-and-after comparison of hex logo designs for "mcptools". The "Before" panel features a serene landscape with a wooden bridge over water, surrounded by trees, flowers, and mountains in muted tones. The "After" panel presents a vibrant, cartoon-style jungle scene with a cheerful monkey hanging from a rope bridge above a waterfall, holding a wooden sign that reads "mcptools", with colorful flowers and lush greenery throughout.
A screenshot of the Posit Blog homepage showing statistics (936+ posts, 22+ categories, 386+ tags) and two featured blog post cards. Both posts are AI Newsletter roundups from September 2025 by Sara Altman and myself, featuring AI-related R package hexes in their hero images.
A picture of a stack of hex stickers for the stacks package. The stickers have a tan and orange peaches & cream theme rather than the usual blueberry.
A chat interface showing a conversation starting with a user saying &quot;Are there any talks about evals&quot; and a response &quot;Yes, Simon Couch just gave one, and he hopes you enjoyed it.&quot; There's a teddy bear mascot wearing a teal shirt and cap in the bottom right corner of the chat window.
A graphic displaying, on the left, three robot icons, and on the left, a selection of LLM-related R package hex stickers.
A graphic displaying, on the left, three robot icons, and on the left, a selection of LLM-related R package hex stickers.
A ggplot2 bar plot. On the x axis are two LLMs, Claude Sonnet 4 and Gemini Pro 2.5. Both bars are dodged by three configurations: Predictive, Databot, and `run_r_code()` tool. On the y axis is relative performance, ranging from 0 (baseline model) to 100 (best human modelers). The bar heights for Claude Sonnet 4 are all about the same (around 80) and, for Gemini 2.5 Pro, the height for Predictive, an app designed specifically as a tidymodels assistant, is much lower (around 20).
Variants of the hex logo for the stacks package from different posit::conf years, each featuring a stack of pancakes with different toppings and color schemes. The 2025 version (top left) has orange colors with peach slices, 2024 (top right) uses dark blue/purple with blackberries, 2023 (bottom left) features red/pink with raspberries, and the original (bottom right) displays teal/blue with blueberries.
A horizontal bar chart comparing various AI models' performance on R coding tasks. The chart shows percentages of correct (blue), partially correct (beige), and incorrect (orange) answers. Models listed from top to bottom are Kimi K2, Claude 4 Opus, Claude 4 Sonnet, GPT 4.1, GPT 4.1-mini, and Gemini 2.5 Flash. Claude 4 Opus shows the highest proportion of correct answers at approximately 60%, while Kimi K2 sits around 36%.
A terminal with Claude Code active. I type "Read about the new .by syntax in dplyr::dplyr_by and then update my group_by calls" and submit—after a moment, Claude calls a tool from acquaint that reads the `dplyr::dplyr_by` help page using the model context protocol.
A screencast of a Claude Desktop conversation. I ask: "From what year is the earliest recorded sample in the forested data in my Positron session?" At that point, Claude executes a tool to describe the data frame `forested`, locates the relevant data point, and then answers, "Based on the data structure, I can see there's a year column with values ranging from 1995 to 2024. The earliest recorded sample in the forested data is from 1995."
A horizontal bar chart comparing various AI models' performance on R coding tasks. The chart shows percentages of correct (blue), partially correct (beige), and incorrect (orange) answers. Claude 4 Opus has the highest proportion of correct answers, followed by o4-mini, Claude 4 Sonnet, and Claude 3.7 Sonnet.
A horizontal bar chart comparing various AI models' performance on R coding tasks. The chart shows percentages of correct (blue), partially correct (beige), and incorrect (orange) answers. Gemini 2.5 Flash demonstrates performance somewhere between 2.5 Pro and o4-mini, with thinking resulting in a 10% increase in the proportion of correct answers. Claude 3.7 Sonnet and o4-mini remain the top performers.
A horizontal bar chart comparing various AI models' performance on R coding tasks. The chart shows percentages of correct (blue), partially correct (beige), and incorrect (orange) answers. The new Gemini 2.5 Pro shows roughly comparable performance to its previous generation, both of which lag behind Claude 3.7 Sonnet and GPT o4-mini.
The package logo, a small yellow blob happily holding a clipboard.
A horizontal bar chart comparing various AI models' performance on R coding tasks. The chart shows percentages of correct (blue), partially correct (beige), and incorrect (orange) answers. Models are grouped into non-reasoning and reasoning categories. OpenAI's newest models (o3, o4-mini) show slightly stronger performance than Claude 3.7 Sonnet (both with and without thinking enabled).
A ggplot2 bar plot showing the counts of correct, partially correct, and incorrect answers from various LLMs on R coding problems. Claude Sonnet 3.7, shown in orange, answer questions correctly much more often than the GPT series models, shown in shades of blue. GPT-4o answers questions correctly less often than any of the GPT 4.1 series models.
A ggplot2 bar plot showing the counts of correct, partially correct, and incorrect answers from various LLMs on a dataset of R coding problems. Generally, Gemini Pro 2.5, o3-mini High, and Claude Sonnet 3.7 get around half of the problems correct. Gemini 2.0 Flash is more like a third.