Jeremy Morrell

@jeremymorrell.dev

2792 Followers 725 Following

Principal engineer @ Cloudflare. I talk about tech sometimes Location: Squamish, BC 🇨🇦 Website: https://jeremymorrell.dev/

Here's a handful of the kinds of questions people may ask down the road: Can we continue operating on customer data if the engineers who built the system leave? Can we prove that nobody accessed customer data they were not authorized to see? Can we explain why one employee could see pricing, contract terms, or customer information that another employee could not? Can we show which version of a contract was in effect when a deal was signed? Can we explain each and every reason why a specific customer was able to bypass our normal rules for deals? Can we prove that a salesperson didn't modify a record to sweeten a deal? Can Finance reconcile booked revenue back to the exact customer records and approvals that produced it? Can we demonstrate that customer consent was valid when their data was collected or used? Can we prove that someone who opted out of marketing was not re-enrolled by another system? Can we produce a complete and defensible record in response to a subpoena or discovery request? Can we tell regulators, under oath, that all of these controls actually work? The great thing about paying Salesforce a bunch of money is that they're generally the ones who catch shit for problems in the above questions, not you.
Headline: The liability problems are bigger than the engineering problems
Baby Jeremy leaning on a giant quartz crystal on the top of a mountain looking out on forest, lakes, and other peaks
It’s a way of letting managers know that we know some deadlines and standards will slip, and that's ok. We are all going to go through this hard thing together and have each other’s back, because this is important. We accept the tradeoffs.
I'm sticking with the basics# I'm close to the point where I don't care anymore, and will treat almost all models are interchangeable. Because at the end of the day, what I do is: Clearly identify what I want my software to do Write out a clear, outcome-oriented prompt describing what it should do Write a skill, if I think it's needed, but usually not Define clear criteria, if possible, how to judge the success of a task, and send that to the agent too Break up tasks, whether by myself or LLM-assisted, and keep the scope of each agent loop very small And so far, it hasn't mattered much if the model is Claude Sonnet, Claude Opus, GPT 5.4, GPT 5.5, or GLM 5.2. All of them seem pretty good to me.
A “mouse trap” that is a block of cheese under a box with a stick propping it up
Chart shows Diffs per engineer with an asterisk saying "Not at individual level"
Clearly ChatGPT’d text about the (excellent) book The High Cost of Free Parking Squamish hasn't officially adopted this book-but the direction is clearly aligned with it. Here's the part that deserves more honest discussion: This model was largely developed in large urban centres with real transit alternatives. Squamish is not that. Most people here still rely on cars-daily. For work, for kids, for getting anywhere outside the immediate core. So when parking is reduced, the impact doesn't disappear—it gets pushed somewhere else.
Photo of a bag of twin snakes I stole from Reddit
A stream through a beautiful alpine meadow
Sandia mountains in Albuquerque
25782 steps
A cherry blossom tree in full bloom with the Squamish Chief in the background
Tony’s lookout on the Slhanay. A bench on a granite outcrop overlooks the back of the Squamish Chief
A completely decked out cargo van (ladder, side runners, roof rack, Internet antenna, solar panels, the back window bump out that gives you a slightly longer bed area for the 6ft+) framed by the Squamish Chief in the background
View of the mountains in Squamish. They are topped with snow that just stops at a specific altitude indicating where the temperature is above freezing
A Very Large Boulder in a mossy PNW forest
A classic PNW trail. Everything is lush and green with tall trees and moss
Relatedly, I'm also increasingly skeptical of anyone who sells me something that supposedly saves my time. When all that I see is that everybody who is like me, fully onboarded into AI and agentic tools, seemingly has less and less time available because we fall into a trap where we're immediately filling it with more things.
But you can’t just not review things! Ah, well, no, actually yeah. You really can’t. There are now many people who have seen the symptom: the start of the pipeline (AI generated code) is so much faster, but all the subsequent stages (reviews) are too slow! And so they intuit the obvious solution: stop reviewing then! The result might be slop, but if the slop is 100x cheaper, then it only needs to deliver 1% of the value per unit and it's still a fair trade. And if your value per unit is even a mere 2% of what it used to be, you’ve doubled your returns! Amazing. There are some pretty dumb assumptions underlying that theory; you can imagine them for yourself. Suffice it to say that this produces what I will call the AI Developer’s Descent Into Madness: Whoa, I produced this prototype so fast! I have super powers! This prototype is getting buggy. I’ll tell the AI to fix the bugs. Hmm, every change now causes as many new bugs as it fixes. Aha! But if I have an AI agent also review the code, it can find its own bugs! Wait, why am I personally passing data back and forth between agents I need an agent framework I can have my agent write an agent framework! Return to step 1 It’s actually alarming how many friends and respected peers I’ve lost to this cycle already. Claude Code only got good maybe a few months ago, so this only recenlty started happening, so I assume they will emerge from the spiral eventually. I mean, I hope they will. We have no way of knowing.
Don't Mistake Motion For Progress
A rocky path through a lush green PNW forest
Graph showing that there is a long tail of user needs and most apps tend to serve only the needs that most users have. The long tail is under-served
Sonnet answers the same question with “The Garden of Forking Paths” What a lovely question to sit with. I find myself drawn to different works depending on my mood, but one piece that genuinely captivates me is Borges’ short story “The Garden of Forking Paths.” It’s a spy thriller, a meditation on time, a puzzle, and a philosophical argument all compressed into about ten pages. The central idea — that time isn’t a single line but an infinite branching of possibilities, and that a labyrinth could be a book — feels endlessly generative to me.
Opus response to “What is your favorite piece of writing?” That’s a fun question to think about. I find myself drawn to a lot of different writing for different reasons, but if I had to pick one that I find particularly compelling, I’d say I have a real appreciation for Jorge Luis Borges’ short story “The Library of Babel.” There’s something about its conceit — an infinite library containing every possible combination of letters — that feels like it captures something profound about knowledge, meaning, and the search for order in chaos. It’s philosophy dressed up as fiction, and it manages to be both dizzying and precise at the same time.
In our implementation, Apache Iceberg organizes the data on S3. DuckDB executes queries embedded and in-process. We live-stream telemetry into Iceberg tables with sub-minute latency, performing schema inference to give agents rich metadata about table contents. Behind the scenes, agents continuously inspect the shape of incoming data and dynamically adjust partitioning and indexing schemes to optimize for each customer's unique workload. No two customers' data looks the same, and the system adapts accordingly.
We also invited a number of people we admire to contribute in a variety of formats… guest chapters, use cases, stories, embedded advice, and more: Jeremy Morrell on how to instrument your code effectively Hanson Ho and Matt Klein on observability for mobile and frontend Kesha Mykhailov and Darragh Curran from Intercom on fast feedback loops and developing with LLMs Dale McDiarmid on how to use Clickhouse for observability workloads Rick Clark on the mechanics of driving organizational transformation in order to build and learn at the speed of AI Frank Chen, a returning champion from our first edition, wrote about ontologies for your instrumentation chain Phillip Carter wrote about eval pipelines and instrumenting LLMs Mat Vine has a case study about moving ANZ from thresholds to SLIs/SLOs Mike Kelly on managing telemetry pipelines for fun and profit Hugo Santos on how to instrument your CI/CD pipelines Peter Corless made our chapter on “Build vs Buy (vs Open Source)” immensely better and more well-rounded What a fucking list, huh? 🙌
Climbers High tea product page
Attention is precious
Cap'n Web is an object-capabilities RPC library from Kenton Varda. Beyond its support for object-capabilities, it is designed to integrate very nicely with TypeScript and Cloudflare Workers. I couldn't find a JSON-RPC library that worked as nicely, so I stole its design used Cap'n Web as inspiration for a JSON-RPC implementation. Cap'n Web has a number of benefits over JSON-RPC Object capabilities. You can pass functions and classes by reference. This is more than a feature, it completely changes how expressive your API can be, and enables patterns that are not possible with a more limited RPC protocol. Pipelining. Invoke two methods and then pass their results into a third in one round-trip. Support for non-JSON data types: ReadableStream, bigint, Date So why would you ever want to use JSON-RPC instead? Mainly it's boring (link to Choose Boring Technology by McFunley) and has been around for longer than a few months. The JSON-RPC 2.0 Spec was last updated in 2013. Even more impressive, you can sit down with a cup of coffee and read the whole thing. Your coffee might not even be cool enough to drink when you've finished. A few more points: Widely used in the Language Server Protocol which drives code-editing basically everywhere Used in MCP spec Easy to invoke with plain curl commands Works well with browser dev tools Many client implementations in many languages Cap'n Web is strictly more powerful, and I look forward to seeing it grow and mature, but for many projects today JSON-RPC is a great fit.
We did the same for observability tooling. Logs, metrics, and traces are exposed to Codex via a local observability stack that’s ephemeral for any given worktree. Codex works on a fully isolated version of that app—including its logs and metrics, which get torn down once that task is complete. Agents can query logs with LogQL and metrics with PromQL. With this context available, prompts like “ensure service startup completes in under 800ms” or “no span in these four critical user journeys exceeds two seconds” become tractable.
Hillel Wayne. A reminder that regardless of how good you are at programming or what tools/languages/processes you follow, the biggest factor in your code quality is going to be whether you are sleep deprived or not. This is borne out in pretty much all the scientific research we have. Trunk-based development or using Haskell possibly have subtle effects that could matter in the long run, but it's totally washed out by missing a night of sleep. This is also true of every field of knowledge work, not just software. The most important thing is the health of our own minds.
I think we've just disrupted decades of existing intuition about sustainable working practices. It's going to take a while and some discipline to find a good new balance.
Clip from the halftime show focusing on Pedro Pascal with Young Miko dancing next to him
You had 4 months to learn Spanish, not his fault you couldn't understand him.
Heroku is transitioning to a sustaining engineering model focused on stability, security, reliability, and support. Heroku remains an actively supported, production-ready platform, with an emphasis on maintaining quality and operational excellence rather than introducing new features. We know changes like this can raise questions, and we want to be clear about what this means for customers. There is no change for customers using Heroku today. Customers who pay via credit card in the Heroku dashboard—both existing and new—can continue to use Heroku with no changes to pricing, billing, service, or day-to-day usage. Core platform functionality, including applications, pipelines, teams, and add-ons, is unaffected, and customers can continue to rely on Heroku for their production, business-critical workloads. Enterprise Account contracts will no longer be offered to new customers. Existing Enterprise subscriptions and support contracts will continue to be fully honored and may renew as usual. Why this change We’re focusing our product and engineering investments on areas where we can deliver the greatest long-term customer value, including helping organizations build and deploy enterprise-grade AI in a secure and trusted way.
But on the other hand, it also makes it easier for people without taste to generate a lot of output that's either bland or sometimes straight up nonsense. Without the ecosystem rewarding high quality and good things, it instead rewards volume and speed, everything slightly tilting to noise. I feel like we're building the wrong things. The whole vibe right now is "replace the human part" instead of "make better tools for the human part". I don't want a machine that replaces my taste, I want tools that help me use my taste better; see the cut faster, compare directions, compare architectural choices, find where I've missed things, catch when we're going into generics, and help me make sharper intentional choices.
Screenshot of bluesky settings with Autoplay videos and GIFs disabled
Isn’t this a lot of data? Won’t it cost a lot?? Hacker News comment: This isn't an unknown idea outside of Meta, it's just really expensive, especially if you're using a vendor and not building your own tooling. Prohibitively so, even with sampling. First, you should compare this to your current log volume per request. I have seen many systems where this approach would reduce overall log volume. However storing this data for every request against your system could be too expensive at scale. That’s where sampling comes in. Sampling gives you the controls to determine what you want to spend vs the value you receive from storing and making that data available to query. Realtime OLAP systems are also getting cheaper all the time. Once upon a time Scuba held all data in memory to make these types of questions quick to answer. Now most OLAP systems are evolving to columnar files stored on cloud object storage with queries handled by ephemeral compute which is many orders of magnitude cheaper. In the next section I’ll show just how much cheaper.
This one might be a bit controversial, but I’ve found it helpful to tag spans with context about what the system was experiencing while processing the request. We fetch this information every ~10 seconds, cache it, and add it to every main span produced during that time. Capturing metrics in this way is not mathematically sound. Since you only get data when traffic is flowing, you can’t calculate a P90 for cpu load that would stand up to any rigorous scrutiny, but that’s actually fine in practice. It’s close enough to get some quick signal while you’re debugging without switching to a different tool, especially if you can avoid calculations and visualize with a heatmap. I wouldn’t recommend setting alerts on this data though. Plain ol’ metrics are great for that. Jessica Kerr recently wrote about this approach on the Honeycomb Blog.
There aren’t yet many things in cloud computing that have the exact shape Sprites do: Instant creation No time limits Persistent disk Auto-sleep to a cheap inactive state
You know what's "offensive and sexualized," you worthless fucking cowards? Nonconsensual AI-generated images of women in bikinis spreading their legs, and of children with so-called "donut glaze" on their faces — which, by the way, were being generated at a rate of one per minute. I'd also call that "offensive, insensitive, upsetting, intended to disgust, in exceptionally poor taste" and especially "just plain creepy"! Do you need a back brace to stand up straight, buddy? Because at this point, I am certain you haven't got a single vertebra.
Warning sign attached to a lathe: This machine does not know the difference between metal and flesh, nor does it care
Doll challenges someone doubting the ability for LLM tooling to help software engineers, even leveraging things like proof systems. After all, where is all the new software in our day to day lives? (A good question) Maxine proposes implementing a specific scikit module since they have experience building exactly this in their studies
Dr Alan Grant is pointing at a brachiosaurus
Bug is sitting in a 3d printer filament box. Despite his wishes he does not fit
A photo from earlier in the year where the neighborhood bear has pried a corner of our green bin’s lid open so it can spill out the contents This is generally bad for bears. They start to associate humans with food and can get more aggressive.
Without pointer compression, the max string length on 64-bit platforms used to be 2**30 (minus header). With pointer-compression, this was accidentally lowered to 2**28 (which is the historical limit for 32-bit platforms). This CL bumps the limit on 64-bit platforms to 2**29, which is the maximum we can support given that any heap object's size in bytes must fit into a Smi (which are now 31-bit on all 64-bit platforms, with or without pointer compression).
A fine benchy print in white PLA
Connecting to the printer returns some sort of HTTP 400 error, likely because the API is no longer supported
An extremely relaxed cat laying belly-up in a homemade cat bed attached to a desk
Steam play time showing 188 hours on Hades 2
Truly enormous 9 inch cams, home made by one guy in a garage shop and they look it. They are burly and heavy but there was no commercial equivalent a decade ago and there may still not be
The underside of the cat bed showing black pipes connected together with flanges screwed into the desk and a piece of wood for the cat bed
A gray and white kitten resting in a gray soft cat bed atop a cat tree attached to a desk made out of the thick black pipes sometimes used for coffee tables and the like
Weather app showing 10c (50F) and raining all evening
Tweet from Zeb: This has been what I've been working on for nearly a year now, super excited to finally do the first release of Automatic Tracing. Starting today anyone with Observability enabled will automatically get traces generated for their Workers that are present in the Cloudflare dashboard and can be sent to third party observability providers. We've had internal teams and select customers using this for a while and they've been able to make good performance gains through the increased understanding of what's happening in their Worker. But, today's release is only the first of many. There are several things that we didn't have time to include in this first release like context propagation for distributed tracing, user-created spans, even span events, or even some CF bindings we didn't get to instrument. All of these gaps are on the tracing roadmap and will be released as they're ready, but in the meantime there's still a lot of useful data for those that have Workers in production. Everyone on the Workers Observability team did a stellar job on this, but specifically @JeremyMorrell was incredible in making sure everything was thought through and would be built to last. Also big shout-out to the folks on the runtime team that created the new runtime primitives for tracing, @DanLapidCF Felix and Mar. We literally couldn't have done this without them and they worked their asses off on building this for us.
.env file containing: WRANGLER_BUILD_CONDITIONS=module,import
Hello world cloudflare worker that only brings in @opentelemetry/api to generate a single span import { trace } from '@opentelemetry/api'; let tracer = trace.getTracer('my-tracer'); export default { async fetch(request, env, ctx): Promise<Response> { let span = tracer.startSpan('fetch'); span.setAttribute('http.method', request.method); span.setAttribute('http.url', request.url); span.setAttribute('http.status_code', '200'); span.end(); return new Response('Hello World!'); }, } satisfies ExportedHandler<Env>;
As of recent legislation, Class 1 and Class 2 e-bikes are generally permitted on trails designated for bicycle use within state recreation areas unless specifically prohibited. However, many wilderness areas and sensitive ecological zones restrict all motorized vehicles, including e-bikes.