Zach Levonian

@zwlevonian.bsky.social

640 Followers 1406 Following

Human–computer interaction researcher. PhD from University of Minnesota. Tacoma, WA. Mastodon: zwlevonian@hci.social

Screenshot of alert email: Dear AWS Customer, You requested that we alert you when the actual cost associated with your org-total-monthly budget exceeds $800.00 for the current month. The month actual cost associated with this budget is $1,989,742,178.79. You can find additional details below and by accessing the AWS Budgets dashboard. A table shows the Alert Threshold as $800. The Actual Amount is $1,989,742,178.79.
Please provide any feedback you would like to share with ACM Leadership below: I support giving automated tools better pathways to access the ACM DL. As an ACM author, I strongly supported the open-access initiative. Better tooling, metadata, and pathways to access further the mission of the ACM. Practically, I publish with ACM venues because they are an effective way to disseminate my work, and so I expect the ACM to add additional access pathways (such as the HTML view) that make the work more accessible. That includes pathways (potentially paid) for LLM-based access. One auxiliary note: the protections required to sustain the ACM DL against unauthorized scrapers also have a significant cost for humans. I am a Wikipedia editor, and I regularly attempt to cite ACM publications. However, the automated citation formatting tool that Wikipedia provides to editors (Citoid) uses Zotero translators, which no longer function correctly on ACM DL URLs! This is a real cost to authors; their work will be less likely to be formatted correctly (with full author info, DOI, etc.). That is part of why I support the creation of alternate pathways that enable appropriate access, including to "public goods" like Wikipedia and Semantic Scholar.
6 cards as a stack, with the bottom card revealed. The same 6 cards in a ring.
The True Believer, patron saint Amanda Askell
Most-liked domains: arxiv.org 223 substack.com 147 github.com 79 simonwillison.net 72 github.io 71 doi.org 52 acm.org 49 nytimes.com 36 huggingface.co 33 youtube.com 33 …and the accounts whose links I've liked the most: @simonwillison.net 62 @davidcrespo.bsky.social 56 @beenwrekt.bsky.social 31 @tedunderwood.com 29 @emollick.bsky.social 28 @gracekind.net 27 @dingdingpeng.the100.ci 27 @timkellogg.me 24 @mariaa.bsky.social 23 @lastpositivist.bsky.social 21
Frequency of GenAI usage by discipline. Humanities & Arts students have a much higher rate of never using genAI. Amusingly, none of the business/finance students say they never use genAI. The underlying percentages and counts are available at https://levon003.github.io/2026/06/13/henricksen-gallagher-genai-survey.html
This strikes me as a common mistake among tech futurists. It is relatively easy to tell a story that says (1) our society has been defined by this one scarce resource. (2) But technology is about to make that resource abundant. This changes everything. (3) Just imagine the wonderful possibilities once this new era of abundance has arrived. They're missing a critical fourth step: (4) With every new abundance, there will be a new scarcity. And how we manage that scarcity will tell us what sort of society we will be living in. Look toward the scarcity, not the abundance.
Model Name Description Aphorism One sentence, but it always feels right Haiku Small poem, small bill Marginalia Provides unprompted commentary on your code Abstract Summarizes reasoning it hasn’t done Sonnet Medium poem, medium bill Diatribe Sonnet, but angry Opus Long poem, entire bill Treatise Opus, but citation is left as an exercise for the reader White Paper Treatise, but you have to enter your email to get it Mythos Opus, but scary Fable Mythos, until the question matters Fable (xhigh) Bankruptcy speedrun Saga Fable, but with extra meandering Saga (Unabridged) Includes answers to unrelated questions Canon Refuses to contradict prior answers Lore Interpretation requires a wiki Cinematic Universe Multiple Sagas with a Canon dispatch layer
Old man saying: When I was growing up, things were better! You didn't have to work hard just to pay bills! You could poop wherever you liked and people would clean it up! And whenever you screamed, someone put boobs in your face, and *food* came out of them. Bottom text: All false nostalgia is essentially the same.
Mount Rainier, from along the Point Ruston shoreline
Meaning of “sage” in French But let’s take a step back. In France, you will hear a lot of discussion around the word “sage“. It approximately translates into well-behaved, good, and wise, all rolled up into one. “As-tu été sage?” Have you been good? Even very young French children will have heard the word so often, that they will be able to nod yes or no to that question. “Sois sage“. Be good. For my own little Frenchies, I usually translate it into English as “smart”. From https://snippetsofparis.com/french-children-sage-or-sauvage/
Societal problem Researchers construct models of social media users to understand human behavior and deliver improved digital services. Such models use conceptual categories arranged in a taxonomy to classify unstructured user text data. Technical problem In many contexts, useful taxonomies can be defined via the incorporation of qualitative findings, a mixed-methods approach that offers the ability to create qualitatively-informed user models. But operationalizing taxonomies from the themes described in qualitative work is non-trivial and has received little explicit focus. Technical solution We propose a process and explore challenges bridging qualitative themes to user models, for both operationalization of themes to taxonomies and the use of these taxonomies in constructing classification models. Method detailsFor classification of new data, we compare common keyword-based approaches to machine learning models. We demonstrate our process through an example in the health domain, constructing two user models tracing cancer patient experience over time in an online health community. How the solution helps society We identify patterns in the model outputs for describing the longitudinal experience of cancer patients and reflect on the use of this process in future research.
Text: With breakthroughs in AI and other emerging technologies, every young learner has the potential to receive high-quality, personalized literacy interventions at the right time. Through its program focused on early literacy outcomes, the Learning Engineering Virtual Institute (LEVI) will spur the development, scaling, and implementation of AI-driven tools designed to reduce the number of struggling readers by half in the early grades.
With captions: https://commons.wikimedia.org/wiki/File:WP25_Anthem_video.webm
Screenshot of table: Hoax Length Creation date Deletion date Size (bytes) Links Jacob Isaacson Fictitious composer 17.49 years June 29, 2008 February 19, 2026 3,722 Deletion discussion, including admission of hoax by creator Archived version of the hoax Rainbow fish (mythology) Fictitious Hindu myth 17.49 years December 13, 2005 June 9, 2023 1,667 Deletion discussion Archived version of the hoax Jay Orpin Fictitious songwriter and producer; claimed to have worked with famous musical acts including the Backstreet Boys, NSYNC, Bon Jovi, Britney Spears, My Chemical Romance, Fall Out Boy, Linkin Park, and numerous others 17.48 years January 2, 2007 June 25, 2024 4,666 Deletion discussion Archived version of the hoax
Screenshot of Wikipedia article: Jacob Isaacson (May 5, 1911 – September 8, 1980) was an American composer and musician. Isaacson was most noted for his own Colortone musical notation and his early works within this system. His association with the Fluxus movement was played down by Isaacson, who held European classical tradition in high regard, although his experimental and minimalist compositions drew inevitable comparison.
Title: Rudeness and Rapport: Insults and Learning Gains in Peer Tutoring Authors:Amy Ogan, Samantha Finkelstein, Erin Walker, Ryan Carlson, Justine Cassell Abstract. For 20 years, researchers have envisioned artificially intelligent learning companions that evolve with their students as they grow and learn. However, while communication theory suggests that positivity decreases over time in relationships, most tutoring systems designed to build rapport with a student remain adamantly polite, and may therefore inadvertently distance the learner from the agent over time. We present an analysis of high school friends interacting in a peer tutoring environment as a step towards designing agents that sustain long-term pedagogical relationships with learners. We find that tu- tees and tutors use different language behaviors: tutees express more playful- ness and face-threat, while tutors attend more to the task. This face-threat by the tutee is associated with increased learning gains for their tutor. Additionally, a small sample of partners who were strangers learned less than friends, and in these dyads increased face-threat was negatively correlated with learning. Our findings support the idea that learning companions should gradually move to- wards playful face-threat as they build relationships with their students.
@zwlevonian, pictured in a cloud of points labeled "Human-Computer Interaction Researchers", adjacent to bubbles for Machine Learning Researchers and AI Scientists, Mathematics Education Leaders, and – strangely – Oregon State Beaver Fans.
Screenshot of text: If working code no longer reliably signals comprehension, education must optimize for something else: the ability to reason about systems – how components interact, where constraints live, and how changes ripple through a codebase. For junior developers, this shift is especially important. As AI accelerates output, leveling up now depends less on writing more code and more on understanding existing systems along with their architectures and embedded trade-offs.
Screenshot of text: The "at least one" in the prompt is deliberately aggressive, and seems likely to force hallucinations in case an article is definitely error-free. So, while the sample here (running the prompt only once against a small set of articles) would still be too small for it, it might be interesting to investigate using this prompt to produce a kind of article quality metric: If it repeatedly results only in invalid error findings (i.e. what a human reviewer Disagrees with), that should indicate that the article is less likely to contain factual errors.
LLM-generated prose undermines a social contract of sorts: absent LLMs, it is presumed that of the reader and the writer, it is the writer that has undertaken the greater intellectual exertion. (That is, it is more work to write than to read!) For the reader, this is important: should they struggle with an idea, they can reasonably assume that the writer themselves understands it — and it is the least a reader can do to labor to make sense of it. If, however, prose is LLM-generated, this social contract becomes ripped up: a reader cannot assume that the writer understands their ideas because they might not so much have read the product of the LLM that they tasked to write it.
Expanding humanity's collective knowledge is, I think, the most meaningful thing we can do. If humans could not usefully contribute to science anymore, this would be a disaster. So, no. I do not think it worth it to find a cure for cancer faster if that means we can never do science again.
Info cards for 10 books checked out from the Internet Archive. The collection of books is eclectic, and consists of mostly 20th century reference works.
Fred Turner: Any whole-system approach doesn’t work. What I would recommend is not that we abandon technology, but that we deal with it as an integrated part of our world, and that we engage it the same way that we engage the highway system, the architecture that supports our buildings, or the way we organize hospitals. The technologies that we’ve developed are infrastructures. We don’t have a language yet for infrastructure as politics. And enough magic still clings to the devices that people are very reluctant to start thinking about them as ordinary as tarmac. But we need to start thinking about them as ordinary as tarmac. And we need to develop institutional settings for thinking about how we want to make our traffic laws. To the extent that technologies enable new collaborations and new communities, more power to them. But let’s be thoughtful about how they function.
Translating Technical Talent into Education Impact What does this support look like in practice? Zach Levonian, a Senior Machine Learning Engineer with Digital Harbor and the LEVI Engineering Hub, has worked closely with the team at Rising Academies to build a data infrastructure and alerting system for their AI-powered math tutor, Rori. This system ensures that Rori can effectively moderate student input and generate safe LLM outputs. “This kind of work requires a broad range of skills,” notes Christie. “It requires back-end development, automated testing, data pipelines, user experience research, and knowledge of the educational domain. It’s difficult to find a single person with this broad of a skillset, but Zach has it.” What makes experts like Levonian even rarer finds is their ability to speak to both technological and educational audiences. Someone may have a strong grasp of natural language processing, but may not have a sense of how to design feedback loops that actually align with student learning needs and progressions. Designed and staffed to support LEVI’s target learning outcome, the Engineering Hub brings that dual perspective.
Bad photo of the Olympia, WA Capitol Dome. The stairs are covered in protesters, and protest signs are visible in the foreground with anti-Trump and anti-DOGE messages.
Text in section 2.2: Generalist RM requires to generate high-quality rewards beyond specific domains (Hendrycks et al., 2021; Jimenez et al., 2024), where the criteria for rewards are more diverse and complex, and there are often no explicit reference or ground truth. To this end, for general domains, we adopt principles to guide reward generation in place of artificial rules. Principles for LLMs are first introduced in Constitutional AI (Bai et al., 2022b; Sharma et al., 2025), which are handicraft criteria that guide the LLMs or curated classifiers to construct safe data pipelines. With principles, the reward generation of GRMs changes to R = C ∼ rθ (x, {y_i}^n_{i=1}, {p_i}^m_{i=1}) where {p_i}^m_{i=1} denotes the principles. We conduct a preliminary experiment to examine the influence of proper principles on reward quality, with the Chat Hard subset of Reward Bench (Lambert et al., 2024) and the IFEval subset of the PPE benchmark (Frick et al., 2025). We used GPT-4o-2024-08-06 to generate the principles and then pointwise rewards four times for each sample. And we filtered the principles whose according rewards are aligned with the ground truth. We tested different LLMs with principles generated by themselves and the filtered principles, and compared them with the default setting with no principle guidance. The results are shown in Table 1. We found that the self-generated principles barely improve performance, but the filtered principles could significantly boost the reward quality. This indicates that proper principles better guide reward generation under correctly summoned criteria. Details are depicted in Appendix D
A research table categorizing dozens of papers into four broad peer characteristic categories: Health (e.g. specific diagnosis), Demographics (e.g. age, gender), Life (e.g. geography, social role), and Other (e.g. Communication style, interests).
6 screenshots show listifications DMs: @sniffer.on.computer has added you to the "a bit stinky" moderation list @jacksmith.is-extremely.gay has added you to the "Smith Family Reunion Apr 16 Be There!" moderation list @stepheny.gaysky.social has added you to the "NSFW / Pro-Ship" moderation list @hogan.swifties.social has added you to the "Porn Bots" moderation list @handle.invalid has added you to the "Block Furry Posts" moderation list @august.musicsky.social has added you to the "Zionist Allies" custom feed list
Does Google use customer data to improve the model(s)? No. Google does not use any of your content (such as documents and predictions) for any purpose except to provide you with the Document AI service. See the Document AI data usage policy. At Google Cloud, we never use customer data to train our Document AI models.
I think of the LLM use here a bit in terms of what my colleague Sam Wineburg calls “taking bearing” — when we land in the middle of a cognitive task like this the impulse is to run straight off towards the first issue we think of. Taking bearing means surveying the landscape before running off into your research session. Part of that is understanding what you’re looking at, whether a source, claim, or set of Google results. But I think part of it is this too, getting something to generate a list of relevant questions, the stuff you might not be thinking about, the things that aren’t part of your initial reaction. You can chose the path yourself, but here’s what’s available. There are usually more directions you can go than you think.
Danny and Lanir set out to give Israeli foreign minister Yigal Allon precise numerical estimates of the likelihood of some very specific bad things happening. They assembled a list of possible “critical events or concerns”: regime change in Jordan, U.S. recognition of the Palestinian Liberation Organization, another full-scale war with Syria, and so on. They then surveyed experts and well-informed observers to establish the likelihood of each event. Among these people, they found a remarkable consensus: there wasn’t a lot of disagreement about the odds. When Danny asked the experts what the effect might be of the failure of Kissinger’s negotiations on the probability of war with Syria, for instance, their answers clustered around “raises the chance of war by 10 percent.” Danny and Lanir then presented their probabilities to Israel’s Foreign Ministry. (“The National Gamble,” they called their report.) Foreign Minister Allon looked at the numbers and said, “Ten percent increase? That is a small difference.” Danny was stunned: if a 10 percent increase in the chances of full-scale war with Syria wasn’t enough to interest Allon in Kissinger’s peace process, how much would it take to turn his head? That number represented the best estimate of the odds. Apparently, the foreign minister didn’t want to rely on the best estimates. He preferred his own internal probability calculator: his gut. “That was the moment I gave up on decision analysis,” said Danny. “No one ever made a decision because of a number. They need a story.”
Diagram from "Burning Diagrams in Anthropology: An Inverse Museum" (Tristan Partridge, 2024)