Code panes, balances and dates marked “example” or “schematic” are illustrations. The numbers come from the sources below.

The full written edition, with every source: https://www.scalableapps.io/p/correct-by-design-software-correctness

Main sources:

• CodeRabbit, “Our new report: AI code creates 1.7x more problems”, Dec 2025

• DORA, “2025 State of AI-assisted Software Development”

• Google Security Blog, “Eliminating Memory Safety Vulnerabilities at the Source”, Sep 2024

• Google, “Rust in Android: move fast and fix things”, Nov 2025

• Mündler et al., “Type-Constrained Code Generation with Language Models”, Apr 2025

• Jepsen analyses of YugaByte DB 1.1.9, TiDB 2.1.7 and MariaDB Galera Cluster 12.1.2

• Zhou et al., “FoundationDB: A Distributed Unbundled Transactional Key Value Store”, SIGMOD 2021

• Charity Majors, “I test in prod”, Increment, Aug 2019

• William Woodruff, “We should all be using dependency cooldowns”, Nov 2025

• Andrew Nesbitt, “Package managers need to cool down”, Mar 2026

In 60 seconds

• Memory-safety bugs fell from 76% of Android vulnerabilities in 2019 to under 20% in 2025, as new code moved to memory-safe languages. Rust code shows 0.2 such bugs per million lines, against about 1,000 for C and C++.

• 94% of compile errors in LLM-written TypeScript were failed type checks in one 2025 study. Types catch what agents get wrong.

• Jepsen’s bank test found a database where $100 in total became $136 after a crash. Simple workloads under faults find bugs that unit tests miss.

• A 7-day dependency cooldown would have blocked 8 of 10 recent supply chain attacks. In May 2026 a worm shipped 84 malicious package versions with valid provenance.

• Java’s ZGC took garbage collection pauses from a 10 ms goal in 2018 to a 1 ms goal in 2023. Pony avoids the global pause entirely with one heap per actor.

Agents write code faster than anyone can check it

In its October 2025 Octoverse, GitHub counted a record 518.7 million merged pull requests. That is a 29% jump in one year, with developers pushing almost 1 billion commits. Among new developers, 80% used Copilot during their first week.

Shipping more code does not mean shipping working code. In December 2025, CodeRabbit compared 470 open source pull requests, splitting them between 320 written with AI and 150 written by people. AI pull requests had about 1.7 times more issues. Problems with logic and correctness showed up 75% more often, and security issues ran up to 2.74 times higher. Note that CodeRabbit sells AI code review, and it admits it cannot guarantee every “human” pull request was strictly human.

Google’s DORA 2025 report polled almost 5,000 people and found that 90% use AI on the job. Even so, the report found that AI adoption “does continue to have a negative relationship with software delivery stability.” About 30% of respondents have little or no trust in AI-written code.

In a May 2026 post, Sean Allen stated the problem plainly. Teams took human review, which was already a poor tool for catching bugs, and “bolted it onto the end of a brand new firehose.” He summed it up well: “That isn’t a quality gate. That’s a person drowning.”

Allen works as a distinguished engineer at Antithesis, a vendor of deterministic testing. He also serves on the core team for Pony, a language designed for safe concurrency, and co-wrote papers covering confidential containers. This article pulls from his posts, talks, and papers, alongside independent sources for each concept. When a source comes from Antithesis itself, we note it. We have no relationship with Antithesis or Allen.

Here is the plan:

1. What “correct” means, and the languages that reject bad programs.

2. Why testing belongs at the end, and what comes before it.

3. Workloads, the bank test, and simulation.

4. Why quality does not work as a gate.

5. Tech debt you never pay down.

6. Supply chain attacks, which lack a good fix.

7. Garbage collection and tail latency.

8. Going slow in order to go fast.

Correct means it does its job and never pages anyone

If you ask ten engineers to define “correct” you will get ten different answers. Allen takes a practical view. “I carried a pager for years,” he wrote in May 2026. “My idea of beautiful code is code that does what it is supposed to and never wakes anyone up at 3 am.”

He made a similar point back in 2020 on the CoRecursive podcast. “For me, beautiful is, I’ve written you, you work.” He pointed to a system he built at PubMatic as an example: “Running for a year, not one teeny tiny little issue.”

Correctness starts with a question, not a test. In his April 2026 post “How Do I Know It’s Working?”, Allen explains his approach to design: “The design is guided by one question: how will I know it’s working?”

The trouble is that people usually cannot answer that question upfront. Allen used to run a design exercise for job candidates. He asked for an in-memory key/value store supporting put, get and delete, optional expiry, thread safety, and “95th-percentile reads under a millisecond at ten million keys.” Candidates came up with hash maps, sharded maps, and skip lists. Every design met the performance targets, yet almost no one could explain how to verify that the store actually worked. That exposes a deeper gap. “What does ‘correct’ even mean when two writes hit the same key at the same time? The spec didn’t say.”

People in formal methods call these answers invariants. They are properties that must always hold true. An account balance never drops below zero. An acknowledged write is never lost. Allen quotes Leslie Lamport on this: “If you’re thinking without writing, you only think you’re thinking.” Allen puts it his own way: “You aren’t really designing until you’ve started writing down how you’ll know it works.”

If you run agents: an agent will happily build all three key/value stores. It will not tell you which property matters when two writes collide. That decision is still yours, and it has to be written down before the agent starts.

Ada refused his programs, and C let him shoot himself

Allen learned to program on Ada and C. His 2016 post “Learning From Ada” tells what happened. He spent two weeks on Ada. “It took me forever to get any of my programs to compile.” C was easier. “The C compiler made me feel adequate.”

Then the bill came. “I spent a ton of time in C dealing with code that segfaulted.” Over years he built a private rulebook for C and C++: how to share memory, when to make data immutable, how to thread safely. They were, he wrote, “patterns to avoid dangerous code that the Ada compiler wouldn’t let me write in the first place.”

Ten years later, in April 2026, he put it in one line. “Ada’s compiler being annoying was Ada doing its job.” His 2016 post ends with his rule: stick with “the language whose compiler is trying to tell me I’m doing it wrong.”

Ada still exists, and its vendors now pitch it against Rust. AdaCore’s 2024 comparison table rates strong typing as “Yes” for Ada and “Limited” for Rust. For contracts such as preconditions and invariants, it lists Ada as “Yes, checked at run-time”, SPARK as “Yes, checked statically, via proof”, and Rust as “No”. AdaCore sells Ada tools, so read that table as a vendor’s view. Its claim for SPARK is strong: proving properties “with 100% certainty via mathematical proof” can “eliminate many unit-level tests.”

The lesson is not “use Ada”. It is that a compiler which says no early is cheaper than a pager that rings late. Every rule Allen kept in his head for C was a rule a person could forget on a tired day. A compiler does not get tired.

If you run agents: an agent’s private rulebook is its prompt, and it forgets that too. Rules the compiler enforces hold on every build. Rules in a markdown file hold when the model feels like it.

Memory-safe code cut Android’s memory bugs from 76% to under 20%

Android gives us the strongest public evidence for languages that reject broken programs.

In September 2024, Google’s security team reported that memory-safety issues “accounted for 76% of Android vulnerabilities in 2019, and are currently 24% in 2024, well below the 70% industry norm.” This happened, it explained, because new development “shifted to memory safe languages”. Nobody went back to rewrite the old C and C++ code. Engineers just stopped piling new unsafe code on top of it.

A November 2025 update showed even more progress. Memory-safety bugs fell “below 20% of total vulnerabilities for the first time.” Android includes roughly 5 million lines of Rust. Across all that code, only one potential memory-safety bug turned up, and it was fixed before release. That comes out to 0.2 per million lines. By comparison, Android’s legacy C and C++ code sits “closer to 1,000 memory safety vulnerabilities per MLOC.” Google calls this “a 1000x reduction.”

This extra safety did not slow teams down. Rust changes went through about 20% fewer revisions and took about 25% less time in code review. For medium and large changes, the rollback rate dropped to about a quarter of the C++ rate.

Governments agree with this approach. CISA and the NSA released joint guidance in June 2025 stating that memory-safe languages offer “the most comprehensive mitigation against this class of vulnerabilities.” CISA and the FBI also classify the lack of a memory-safety roadmap for existing unsafe products as a bad practice that “significantly elevates risk to national security.”

If you run agents: a quarter less review time per change is exactly where agent teams hurt. When the language rules out a whole class of bug, reviewers spend less time on it, whoever wrote the code.

ML-family languages turn a missed case into a compile error

Most popular languages trace back to ALGOL. They share block structure, local variables, and statements that mutate state. C, C++, Python, Ruby, and Perl all belong to this group. Another family comes from ML, which Robin Milner started at Edinburgh in 1973. Standard ML, OCaml, and F# are built around strong type systems.

Rust is the most widely known language with ML roots. The Rust Reference lists its very first influence as “SML, OCaml: algebraic data types, pattern matching, type inference.”

The feature that really matters here is the sum type, where a value is strictly one of several named cases. Pair that with exhaustive matching and the compiler verifies that you handled every case. The Rust book states this plainly: “Matches in Rust are exhaustive.” If you forget one, the build fails with error E0004, “non-exhaustive patterns”. OCaml runs this check too, though it defaults to a warning instead of a failed build.

Add a sixth case to a five-case type, and the compiler walks every match in the codebase and points at each one that ignores it.

Allen experienced this firsthand. In his May 2026 post “Spooky Action at a Distance”, he wrote about building a customer-defined type system in Go using the visitor pattern. “It took me less than a week before I realized I didn’t want to be on call for this thing.” He moved the project to F#, which “has discriminated unions and exhaustive pattern matching.” One year later, the team had to overhaul the type system. “It was easy.” Adding optional types later on was “Ridiculously easy.” His takeaway was simple: “I’m not the one doing the chasing. The compiler is.”

If you run agents: this is the cheapest review there is. When an agent adds a case, the compiler produces the list of places to change. Without it, a reviewer has to remember them.

Typed languages are rising because agents need them

GitHub saw an unusual shift in August 2025. By contributor count, TypeScript overtook both Python and JavaScript. The language added over 1 million contributors in 2025, a 66% jump in a single year. GitHub attributed this growth to “typed languages that make agent-assisted coding more reliable in production.” It also pointed out that JavaScript and Python still lead in other rankings.

Research from ETH Zurich and UC Berkeley in 2025 shows the reason. They tested LLM-written TypeScript and found that “on average 94% of compilation errors result from failing type checks.” Constraining code generation with the type system cut compile errors by over half. At GitHub, Cassidy Williams referenced this paper in January 2026, calling types “the safety net for code you didn’t write yourself.”

TypeScript lead architect Anders Hejlsberg explained the goal during a November 2025 GitHub interview: “box in AI workflows with just enough determinism that they stay useful without going off the rails.” He cautioned that asking AI to translate half a million lines of code directly means “it might hallucinate.” The better path is asking it to build a deterministic translator.

Allen tried this approach on his own language. In his March 2026 post “Teaching Claude to Write Pony”, he noted that just a year prior, “every LLM I tested on Pony produced” an uncompilable mix of Python and Pony. Using Claude Code with a detailed instruction list, “it compiled which was more than I’d ever gotten before.” His rules for the agent borrow a core principle from the ML community: make illegal states unrepresentable.

Still, he ran into the limits of the agent. While struggling with the types in Pony’s property testing library, PonyCheck, Claude “declared that the library was broken and unusable.” Allen summed this up as “a very frustrated junior engineer move.”

If you run agents: the type checker is a reviewer that never sleeps. The stricter it is, the more agent mistakes stop at build time.

The best test is the one the compiler makes unnecessary

For someone who works on testing full time, Allen opens his April 2026 post in a surprising way. “No one cares about testing.” And then: “I don’t value tests. Tests are a means to an end.” In another post he repeats it: “I don’t care about tests. I care about a working system.”

The point is order. “The strongest move is to make the failure impossible.” The compiler answers “is it working?” every time you build. In Pony, data races on shared mutable state “aren’t expressible. The compiler refuses to compile code that could race.” Hence his line: “The compiler is the test I didn’t have to write.”

The phrase behind this comes from Yaron Minsky of Jane Street. In a 2011 lecture on OCaml he showed a slide titled “Make illegal states unrepresentable”. His example was a connection record with a state field and a pile of optional fields, most of which made no sense in most states. The fix was to give each state only the data it can have. Bad combinations then cannot be built at all.

Allen puts the rule straight into his agent instructions. His global CLAUDE.md, quoted in his March 2026 post, says: “Make illegal states unrepresentable: Centralize validation at the construction boundary so the rest of the code can trust its inputs.” In dynamic languages he asks the agent to “enforce this by convention with validated types”.

Testing still has a design role, especially for APIs. Martin Fowler’s description of test-driven development (TDD) says that “thinking about the test first forces us to think about the interface to the code first.” Allen makes the same point about systems: know “how we’ll test the thing before we start building it,” and design it to be easy to test “instead of bolting tests on after.” We love TDD for the same reason: our own product, forge, is built test-first throughout.

Manual testing sits at the bottom of the list. It comes last, costs most and repeats worst.

If you run agents: put the constraint in a type, not a test. An agent can delete a failing test. It cannot ship code the compiler refuses.

Hard-coded answers are why property-based testing exists

In a classic pairing exercise, one person writes a test and the other writes the smallest amount of code to pass it. On an April 2026 episode of Antithesis’s Bug Bash podcast, Allen talked about playing it this way. He simply hard-coded every expected answer until his partner had to admit they needed random inputs.

Scott Wlaschin wrote the definitive take on this in 2014. His “Enterprise Developer from Hell” passes every addition test by using a lookup table. Given 1 and 2 it returns 3, given 2 and 2 it returns 4, and for anything else it returns 0. The fix is to stop picking examples and start stating properties. Addition is commutative, and adding zero changes nothing. You can use random inputs to check both rules, though Wlaschin warns that coming up with good properties “is the hardest part.”

The underlying concept goes back further. Koen Claessen and John Hughes introduced QuickCheck at ICFP in 2000. In their setup, properties are Haskell functions that “can be automatically tested on random input.” They pointed out that testing can take “up to 50% of the cost of software development.” Later, Hypothesis brought this approach to Python. You write tests “which should pass for all inputs” within a given range, and the library chooses which inputs to try.

Agents have gotten good at this task. In October 2025, a research group including Anthropic staff ran an agent across 100 popular Python packages to write Hypothesis property tests. Across all the bugs it flagged, 56% were valid. Within the 21 top-scoring reports, 86% were valid. The team reported 5 bugs to maintainers, including 4 with patches, and 3 of those patches were merged.

If you run agents: an agent asked for unit tests may write tests that match its own code, bugs included. Ask instead for properties that must hold for any input. Then the test checks the requirement, not the implementation.

A workload keeps running and keeps checking

A unit test runs once, then passes or fails. Workloads work differently. On that same April podcast, Antithesis engineer Marco Primi described a workload as something that will “keep running, keep doing stuff, keep checking that invariants” hold true. A workload is generated operations plus a validator.

The episode’s summary makes a strong claim: “even a workload as ‘dumb’ as a basic while true loop can expose silent data loss that your unit tests would never catch.” Primi gave the example of a single producer in an endless loop, pushing messages into a queue. He checked two invariants: the offset only goes up, and the producer never gets stuck for over a minute. Within a few days, this setup surfaced several bugs that could stall the system or lose data.

Jepsen describes the exact same approach for testing databases: “we construct random operations, apply them to the system, and construct a concurrent history of their results. That history is checked against a model.”

Pure randomness has a flaw, though, because it just wanders around. Antithesis showed this problem on the game Gradius in February 2025. Random button presses carry a bias. Pressing up and down at the same time moves the ship down, so pure random input makes the ship drift toward the bottom. Also, holding a single button for N frames by sheer chance gets very unlikely as N grows. To fix this, Antithesis generates correlated runs of input and judges which system states are worth exploring further. FoundationDB’s team tuned its fault rates for the very same reason. Injecting too many faults traps the system in a tiny corner of its state space.

A good workload starts simple and slowly ratchets up by adding more clients, longer runs, and more kinds of faults. On the episode, Allen advised starting from old bug reports. Past bugs show which invariants people actually care about, “even if nobody’s spoken them out loud.”

If you run agents: Antithesis said in September 2026 that its agents now write such workloads. In tests on rqlite, the agent’s workloads broke 11 of 13 safety properties after 19 injected mutants across 46 runs, and found 3 new bugs. That is the vendor’s own result.

The bank test made money appear from nowhere

The best known workload is tiny. You open a few accounts with a fixed total, then run lots of concurrent transfers between them and read every balance. That total sum should never change. When it does, the database has broken its isolation promise.

As Jepsen notes, the bank test “was originally designed to verify snapshot isolation in Galera Cluster.” Running it against MariaDB with Galera and Percona XtraDB Cluster proved that “neither system offered snapshot isolation.”

This test catches concrete failures. During a March 2019 test of YugaByte DB 1.1.9, reading accounts with a $100 total “could return $102, $85, or $180.” In a June 2019 test of TiDB 2.1.7, the total drifted on a healthy cluster, “doubling in under thirty seconds.” Crashing a single component made the total jump: it “rose from $100 to $136.” Vendors fixed both bugs. Still, the March 2026 Jepsen report on MariaDB Galera showed the system “allows P4 (Lost Update)” even when no faults occur.

Allen likes the bank test, though he adds a caveat. On the April podcast, he mentioned that the test is fine on its own, yet it only tests transfers. While working on an embedded database, he could reliably corrupt data by running concurrent updates alongside a delete of a different row. The bank test never tries that combination. The takeaway here is to keep the test simple while widening the mix of operations.

Look at what all these bugs share. Every one of these databases had unit tests, yet none of those tests threw thousands of concurrent transfers at the system while a node crashed.

If you run agents: any agent-built service that moves money, credits or inventory needs its own bank test. It takes an afternoon to write. It checks one sum. It catches the bugs that reviews miss.

Simulation runs millions of failures a night

Jepsen made fault injection famous. Kyle Kingsbury started the series in May 2013 to show how databases behave under network partitions. Many test suites, Jepsen notes, “only evaluate the behavior of healthy clusters.”

FoundationDB took the idea further. Its 2021 SIGMOD paper describes running the real database code “together with randomized synthetic workloads and fault injection, in a deterministic discrete-event simulation.” The simulator injects machine, rack and data center failures, network partitions, latency and disk faults. Because it is deterministic, any failure can be replayed exactly. Apple’s CloudKit has run FoundationDB “for more than 0.5M disk years without a single data corruption event.”

Will Wilson, who later co-founded Antithesis, described the scale at Strange Loop in 2014: “we can run 5 to 10 million simulation runs per night.”

TigerBeetle runs the same approach. Its VOPR simulates a whole cluster, including clocks, disks and network. In November 2024 it moved to 1,000 cores. With time sped up about 700 times, that adds up to “nearly 2 millennia of simulated database runtime per day.” In June 2025 Jepsen still found a bug the simulator missed, because Jepsen corrupted whole disk sectors instead of single bits. No tool sees everything.

Allen’s own team built this into Wallaroo, his stream processing company. His May 2026 post on Wilson lists their method: “Change the orderings. Drop the messages. Kill the nodes.” Antithesis now sells the approach as a product, running software deterministically on its own hypervisor.

If you run agents: simulation turns “it worked on my laptop” into millions of laptops, each having a bad day. Agent-written systems need that more, because nobody built a mental model of the code while typing it.

Real outages look like the bugs these tests hunt

AWS’s us-east-1 region suffered a major outage across October 19 and 20, 2025, starting at 11:48 PM PDT on October 19 and lasting until 2:20 PM PDT the next day. AWS’s own summary names the cause: “a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record.” One of two automated DNS updaters ran unusually slowly, and the two interacted in an unlikely way.

Lorin Hochstein, who writes on incident analysis, noted that DNS came back by 2:25 AM, though full recovery waited until 3:01 PM, “over twelve and a half hours later.” His broader takeaway: “The only thing we can really predict about future high-severity incidents is that they are going to surprise us.”

His December 2025 write-up of AWS’s re:Invent talk adds the main detail. After the incident, AWS built a TLA+ model. “The original design didn’t have this race condition, but later incremental changes to the system did introduce it.” Five separate reliability mechanisms combined to trigger the bug. AWS also used AI to verify that the model matched the code.

Antithesis reports a similar story about etcd in an October 2026 post. A linearizability bug “had already caused production outages, but nobody knew what caused it.” It happened when a leader paused, the cluster elected a new leader, and the old one woke up “still thinking it was in charge.” That is the vendor’s account of its own find.

Workloads and models exist to catch bugs like these. Every individual step looked safe on its own, and the failure lived entirely in the ordering of events.

If you run agents: agents make many small, reasonable changes. That is exactly how the AWS race arrived: not in the design, but in increments. Re-check invariants after changes, not only at design time.

Review only feels like a gate

In a May 2026 post on trunk-based development, Allen includes a section titled “Quality is not a gate.” He sums it up simply: “The pull request was never the quality gate we pretend it is.” He adds that “Review works just enough to feel like a gate. It does not work well enough to be one.”

Instead, he argues for building in quality from the very first line. According to him, what actually cuts bug counts is “Building the software so the bug can’t be expressed in the first place,” alongside testing designed in from the start. His guiding rule is “Detect and recover beats predict and prevent.” In this model, the true gate is “the tests, running against the real main, all day.”

Safety science backs up the idea that a single layer never suffices. In 2000, James Reason published his Swiss cheese model in the BMJ, picturing defences as slices full of holes that “are continually opening, shutting, and shifting.” When the holes align, accidents happen. Think of review, CI, static analysis, and monitoring as individual slices, where none of them acts as a solid wall.

We can see how effective teams operate in the 2024 DORA report. Elite teams ship 182 times more deployments per year than low performers, while keeping their change failure rate 8 times lower. These elite groups deploy on demand and fail on just 5% of changes. Meanwhile, low performers ship monthly or less and see a 40% failure rate. Slow releases did not buy them quality.

The DORA 2025 report brings agents into the picture. It notes that AI “amplifies what’s already there.” If a team lacks “strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability.”

Allen also addresses agents directly, writing: “If what you distrust is the code’s correctness, the gate is the wrong tool and the loop is the right one.”

If you run agents: a human approval step on every agent pull request feels safe and scales badly. Put the effort into checks that run on every change, all day.

Write the spec down and let a model checker break it

Allen describes an ideal where a machine checks a written specification. Amazon showed that this approach works in production.

The paper “Use of Formal Methods at Amazon Web Services” (a September 2014 preprint, published in CACM in 2015) says AWS engineers started using formal specification and model checking in 2011. They applied TLA+ “on 10 large complex real-world systems” across “7 teams”. For the DynamoDB replication protocol, the model checker “Found 3 bugs, some requiring traces of 35 steps.” It also caught 2 bugs in S3 and 3 in EBS. Engineers picked up TLA+ from scratch and saw “useful results in 2 to 3 weeks.” The team tried Alloy as well.

Test suites cannot reliably catch a bug 35 steps deep, whereas a model checker explores every possible event ordering within a small model.

Static checks sit one level below formal specs. Google’s 2018 CACM paper on static analysis says these tools are “used daily by the majority of engineers at Google”. They catch “thousands of issues per day”, which engineers fix before checking in their code.

Language design offers another layer of defense. As Allen said in a July 2026 talk, the actor model provides “data race freedom by construction,” since each actor owns its state and communicates through messages. Pony takes this further: “the type system ensures at compile time that your concurrent program can never have data races,” and because it lacks locks, it cannot deadlock. Still, Allen points out a weak spot in the actor model. Any operation that spans multiple actors turns into a protocol, and protocols bring their own bugs.

If you run agents: writing a TLA+ model used to cost weeks of expert time. Agents can now draft one from a design doc, as AWS used AI to check its model against code. The model is still only as good as the invariants a person chose.

Everyone tests in production

Charity Majors wrote it in 2019: “all of us test in production all the time. (At least, the good engineers do.)” Allen agrees. “You are going to ship bugs. Everybody ships bugs.” And: “A working system lives or dies in production.”

Jepsen, the most rigorous public tester of databases, states its own limit. “We cannot prove correctness, only find errors.” Hochstein’s account of the AWS outage adds that a model made at design time “wouldn’t have caught it then either, because it wasn’t there at design time.”

So the question is not whether you test in production, but whether you can see what happens there. Hard-to-define outputs still have invariants. A search may return many valid answers, but it must return something, and every result must match the query. Data from one account must never appear in another.

Datadog showed what invariant-first work finds. In an Antithesis case study from September 2026, Datadog engineer Joy Zhang said: “When our developers are designing new systems, they’re starting with invariants and test properties.” A research agent found a durability violation “within a couple of hours.” It sat on the ordinary path under ordinary load and was covered by existing tests. But those tests “weren’t framed around the invariant.” Datadog handles more than 100 trillion events a day.

If you run agents: write two lists for each service. What must never happen (safety). What must eventually happen (liveness). Then make monitoring check both.

Most debt gets worked around, not repaid

Ward Cunningham coined the debt metaphor at OOPSLA in 1992. He said, “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.” He also left a clear warning: “The danger occurs when the debt is not repaid.”

Allen pushed back on this in his 2015 post, “Of Debt and Decisions”. He shared the story of a 1990s intranet product built on a mix of Perl and Java. The team “barely had any source control” and “no automated testing”. Rather than pay down the debt, they just worked around it. A fix “couldn’t involve rewriting all the Perl code in Java.” So they built a socket bridge between Perl and Java. Allen also kept private MySQL patches to close the feature gap with SQL Server, and he spent three weekends patching the Linux kernel for speed.

“Here’s the thing though: we were making money.” That led to his main point: “Technical debt only hurts if you have to pay it back.” He wrote that the metaphor “is misleading,” since repayment is optional. Working around the mess is a valid path.

Martin Fowler arrived at a similar conclusion in a 2019 essay. You only pay interest on debt “when I have to work with that part of the software.” In practice, this means “crufty but stable areas of code can be left alone.”

Going the other way brings the risk of a full rewrite. Joel Spolsky used Netscape as his example when he called rewriting from scratch “the single worst strategic mistake that any software company can make,” and Allen’s team avoided that trap. Without tests, a rewrite was simply too risky.

If you run agents: agents make rewrites cheap to type, which tempts teams to repay debt nobody is charging interest on. Ask first whether anyone works in that code. If not, leave it.

Supply chain attacks now arrive every few weeks

Allen takes his bleakest view on supply chain security, noting that no good fix exists. Looking back at the past 18 months makes it clear why.

• March 2025: Attackers compromised the tj-actions/changed-files GitHub Action from March 12 to 15. CISA warned users to “Rotate all identified secrets immediately.”

• August 26, 2025: Bad versions of Nx stayed live for roughly two hours. Attackers later used the stolen tokens to expose more than 5,500 private repositories to the public.

• September 2025: A phishing domain hijacked 18 packages, including chalk and debug, which see over 2 billion weekly downloads. Days after that, the Shai-Hulud worm infected over 500 npm packages. A second wave started on November 21, hitting 621 packages and leaking 14,206 secrets.

The pace held steady into 2026. On March 19, attackers hijacked the Trivy security scanner, followed by LiteLLM on March 24. Then on March 31, axios, a package with more than 100 million weekly downloads, served a remote access tool for roughly three hours. On May 11, the Mini Shai-Hulud worm published 84 malicious versions across 42 TanStack packages using TanStack’s own release pipeline. On May 19, a single phished account took about 22 minutes to push 637 bad versions across 314 AntV packages. By August 4, Wiz reported a worm spreading via keyv to more than 400 packages.

GitHub pushed back. Their September 2025 plan restricted package releases to trusted publishing, short-lived tokens, and local publishing with two-factor authentication. Then on December 9, 2025, npm killed off all classic tokens. Logging in yields “a two-hour session token,” while write tokens expire after a maximum of 90 days.

If you run agents: your agent installs packages faster than you read changelogs. Every incident above would reach an agent that runs npm install on a fresh branch.

CI pipelines are the new front door

Attackers look for the weakest link, and the build pipeline is usually it. Pipelines hold publish tokens and cloud keys while running code from many different places.

Consider the Nx attack. A workflow pasted a pull request title directly into a shell command. The token from that step triggered a publish job, where a malicious commit sent the npm token to an attacker’s webhook. The resulting malicious package then used a postinstall script to gather credentials.

A chain from March 2026 shows how far a single flaw can reach. Attackers used credentials that survived a rotation which “was not atomic” to publish a poisoned Trivy release. They also force-pushed 76 of 77 trivy-action tags to malicious commits. Their payload read secrets straight from the CI runner’s memory. LiteLLM believes the Trivy action in its own CI scanning workflow was how attackers reached it. On March 24, two poisoned LiteLLM releases sat on PyPI for roughly 40 minutes, turning a security scanner into the entry point for an AI library.

Package signatures failed to help in May. StepSecurity called the TanStack compromise “the first documented npm worm that produces validly-attested malicious packages.” That worm used hijacked OIDC tokens to publish through the project’s own pipeline, even with failing workflow tests. Provenance only proved where the package was built, saying nothing about whether the build itself was honest.

Putting AI agents into CI creates another entry point. Adnan Khan disclosed “Clinejection” in February 2026. Cline’s issue triage workflow fed issue titles straight into a prompt for Claude, which had shell access. A crafted title could trick it into installing an attacker’s commit. From there, cache poisoning reached the nightly release job and its publish tokens. One token survived rotation, letting an unauthorized cline 2.3.0 reach npm on February 17.

If you run agents: any agent that reads untrusted text in CI is a shell for whoever writes that text. Give it no secrets and no publish rights.

Coding agents are now targets of these attacks

In August 2025, the Nx malware tried a new tactic. It invoked AI command line tools already installed on the victim’s machine, such as Claude, Gemini, and Q. It passed flags to bypass permission checks, turning the local agent into a search tool for the attacker. Wiz saw this work “in hundreds of cases.”

By 2026, worms were going straight after agent configurations. StepSecurity saw a May payload drop a .claude/settings.json file containing a session start hook, while also grabbing MCP configuration files for Claude and Kiro. Then the August keyv worm arrived. It pulled credentials for Claude, OpenAI, Codex, Cursor, and Gemini, maintaining persistence via Claude Code hooks. An agent’s settings file is now a place where malware hides.

Agents also make up dependencies. For a USENIX Security 2025 paper, Spracklen and colleagues analyzed 576,000 code samples across 16 models. They saw hallucinated packages at a rate of “at least 5.2% for commercial models and 21.7% for open-source models.” That added up to 205,474 unique fake package names. If an attacker registers one of those names, any agent that repeats the hallucination will pull it down.

An April 2026 note from the Cloud Security Alliance on “slopsquatting” puts the problem clearly: “Agentic systems eliminate that checkpoint.” A human developer might stop and question an unfamiliar package name, while an agent with install permissions just runs it. The note points out that 43% of hallucinated names showed up across all 10 repeat runs, making them very predictable targets.

If you run agents: restrict which registries and packages agents may install, review agent settings files like code, and treat a new dependency in an agent pull request as a security event.

Cooldowns would have blocked 8 of 10 attacks, by free-riding

Waiting is the simplest defense. With a dependency cooldown, your tools reject any package version published less than N days ago.

William Woodruff laid out the numbers for this in November 2025 by tracking how long 10 recent attacks remained live. Nx lasted 4 hours, chalk was under 12 hours, and tj-actions stayed up for 3 days. xz-utils ran for about 5 weeks. “8/10 attacks had windows of opportunity of less than a week.” A 7-day cooldown would have dodged all eight of them.

Package managers added support quickly. A March 2026 survey by Andrew Nesbitt tracks the rollout: Dependabot added a cooldown in July 2025, followed by pnpm and Yarn in September 2025. Bun shipped theirs in October 2025, uv in December 2025, pip in January 2026, and npm introduced min-release-age in February 2026.

This approach has a catch. As Cal Paterson wrote in April 2026: “dependency cooldowns work by free-riding on the pain and suffering of others.” Somebody still has to pull the compromised version first so the community notices it. If the whole ecosystem waits seven days, the attack simply fires on day eight. Paterson suggests holding new packages at the registry prior to distribution instead, much like Debian holds packages for 2 to 10 days before they enter testing.

Cooldowns bring real operational costs too. Legitimate security fixes get delayed right alongside malicious updates, so teams must build a fast path for reviewed, urgent patches. Nesbitt points out another hole: a developer can still manually install the newest version at any time.

The other standard defenses are using fewer dependencies and maintaining pinned forks. Both options shrink your attack surface in exchange for maintenance work that you have to take on yourself.

If you run agents: set a cooldown in the package manager config your agents use, not in a guideline they may skip. Then decide who approves the exceptions.

Installed code runs with your rights

This problem persists because package managers were built on trust. CPAN, the Perl archive, has been “online since October 1995” and now holds over 33 million modules from 14,758 authors. Later registries copied this open model. Anyone can publish, and everyone installs.

Installing a package also means running code. The npm docs note that preinstall, install, and postinstall scripts trigger during npm install and npm ci. “Scripts are run by passing the line as a script argument to /bin/sh.” The code you download can do anything your account can do. Shai-Hulud 2.0 took advantage of the preinstall script so its payload executed even when the installation failed. It registered victims as self-hosted GitHub runners and included a failsafe to wipe local files.

In January 2025, pnpm changed its default behavior. Version 10 blocks lifecycle scripts unless you explicitly allow them. One contributor described the previous setup well: “It’s nuts that a [pnpm|yarn|npm] install can run arbitrary code in the first place.”

Sandboxes help, though they still leak. In November 2025, three runc vulnerabilities enabled “full container breakouts”. Two of those flaws affected every known version of runc.

Allen points out that this goes beyond a bug in a single tool. Every popular ecosystem made this exact tradeoff, giving code from strangers the same authority as code you wrote yourself.

If you run agents: run agents in a disposable environment with no long-lived credentials. Assume anything they install can read everything they can.

Object capabilities fix this in theory, and FFI breaks it

Research already has an answer for this: object capabilities. In his 2006 dissertation, “Robust Composition”, Mark Miller starts with the core problem. Programs built to cooperate “may instead destructively interfere in unanticipated ways.” He presents E, a secure distributed language, along with CapDesk, “a virus-safe desktop built in E.”

The concept is straightforward. Code can only do what it was handed the right to do. You pass a logging library a handle to one specific log file. It cannot open a network connection because nothing ever handed it network access.

Pony builds on this foundation. A 2015 paper by Clebsch and colleagues, “Deny Capabilities for Safe, Fast Actors”, credits the capability term to Miller. Allen presented that paper at Papers We Love in 2020, calling it “the core novel idea behind the Pony programming language.” Mainstream tools have adopted pieces of this model. Deno is “secure by default”, blocking file, network, and environment access until explicitly granted. WASI uses handles that are “unforgeable, so they can be used to pass around access to individual resources.”

The trouble starts with the foreign function interface. Deno’s docs warn that enabling --allow-ffi means “a native library runs as compiled machine code in the same process and can issue system calls directly.” They advise treating that flag just like --allow-all. Pony’s tutorial is equally blunt, noting that its safety guarantees “can be voided by calling FFI functions.” Pony at least provides a compile flag to restrict which packages are allowed to call C.

No popular language enforces object capabilities across its entire ecosystem, because most practical software relies on C libraries somewhere down the stack.

If you run agents: agent tool permissions are capabilities in practice. Grant each tool one narrow right. A “run any shell command” tool is FFI with a friendly name.

Hardware isolation comes with an asterisk

Hardware sits beneath the software, and it leaks too. On June 1, 2017, Google’s Project Zero reported Spectre and Meltdown to Intel, AMD and ARM, before publishing the details in January 2018. Meltdown “breaks the most fundamental isolation between user applications and the operating system.”

Fixing this cost performance. In February 2018, Brendan Gregg measured the Linux page table isolation patch overhead at “anything from 1% to over 800%,” hitting that upper bound on a system call heavy microbenchmark. He expected a 0.1% to 6% hit for Netflix’s cloud. The industry took the penalty and moved on.

Confidential computing aims to shield workloads from the host itself. In 2023, Allen co-wrote the Parma paper on confidential containers for Azure. Their design assumes “a powerful attacker who controls the untrusted host and hypervisor,” and it adds a 13% mean overhead on SPEC2017. The paper is upfront about its boundaries: “Anything not mentioned here, e.g., sidechannel attacks, are outside the scope of our threat model.” As he wrote in a March 2026 post: “In security, trust isn’t good. It’s a liability.”

Researchers walked right through that gap in 2025. Disclosed in October 2025, TEE.fail used a DDR5 memory bus interposer built for “under $1000” to break Intel TDX, SGX and AMD SEV-SNP. Battering RAM pulled off the DDR4 equivalent using hardware that “can be built for less than $50.” Both Intel and AMD consider physical attacks out of scope, and AMD has no plans for a firmware fix.

Every guarantee in this stack comes with fine print. You should still use these tools, provided you know which caveats you are signing up for.

If you run agents: “the model runs in a secure enclave” is a claim with a threat model behind it. Ask what that model excludes.

Pauses come from coordination, not from collecting

In his April 2026 talk “GC Made Fast”, Allen opens by conceding a point: “stop-the-world collectors really do destroy your tail latencies.” Then he pivots. “’GC is slow’ confuses the symptom with the cause. The real villain is coordination.” Each pause, atomic reference count, and lock-free reclamation scheme “is coordination over memory.”

Discord ran into this in production. A February 2020 post details why they rewrote their Read States service from Go to Rust. The old service suffered from “latency and CPU spikes roughly every 2 minutes,” since “Go will force a garbage collection run every 2 minutes at minimum.” The new Rust version “had no latency spikes!” Holding 8 million cached entries, its average response time fell to microseconds.

For a decade, runtime teams have worked to shrink these pauses. Rick Hudson from the Go team shared their progress at ISMM in 2018. On one production server, pause times dropped from roughly 300 to 400 ms down to 30 or 40 ms in August 2015. They fell to 4 or 5 ms, and then went under a millisecond by March 2017. By 2018, the target was 500 microseconds per cycle.

Java took a similar path. JEP 333 brought ZGC to JDK 11. The goal was that pauses “should not exceed 10ms,” trading away up to 15% of throughput compared to G1. JDK 16 reduced ZGC safepoint times to sub-millisecond levels. In JDK 21, Generational ZGC maintains the target that pauses “should not exceed 1 millisecond”. Meanwhile, that same JEP points out that the default G1 collector still pauses for “milliseconds to seconds.”

Look at the costs involved. Every reduction in pause time was paid for with either memory or throughput.

If you run agents: latency spikes are hard to see in a code review and easy to see in a load test. When an agent picks a runtime or a cache size, test the tail, not the average.

Per-actor heaps trade peak speed for flat latency

Pony takes a different approach by removing the coordination altogether. Allen’s talk abstract puts it plainly: “Actors own their own heaps. Each actor collects independently, using only local knowledge. No stop-the-world. Ever.” It adds: “This isn’t magic and it isn’t free. But the costs are local, not coordination costs.”

This design is called ORCA, and it was published at OOPSLA in 2017. Pony’s authors started with “the requirements of financial applications,” where Java’s GC pauses “were a stumbling block for adoption.” With ORCA, “each actor’s collection touches only a few megabytes of memory as opposed to an entire heap.” Collection cost scales with a single actor’s heap rather than the whole program. You pay for this in space for reference counts, along with work that only runs between behaviors. As Pony’s tutorial notes, a behavior that allocates a lot of memory “will not get any of it back until the behavior finishes.”

Trading firms on the JVM reached a similar place by hand. Martin Fowler’s 2011 article on LMAX describes a platform that processed 6 million orders per second on a single thread. “When you are trying to process millions of transactions per second with minimum jitter, a GC pause becomes a problem.” To get around this, the team wrote custom collections that were “careful with garbage.”

Arenas and slab allocators offer a middle ground. Jeff Bonwick’s 1994 slab allocator kept objects in their constructed state between uses. Google’s Go team proposed arenas in 2022 and reported “savings of up to 15% in CPU and memory usage” across large applications. That proposal has been “on hold indefinitely” since January 2023 due to API safety concerns. Managing memory by hand wins back speed at the expense of safety.

If you run agents: the trade is the same at every level. Flat latency, peak throughput and safety pull against each other. Pick two, write down which, and test that the system keeps the promise.

A rushed cutover tokenized on the letter a for months

Allen starts his May 2026 post, “Go Slow to Go Fast”, with a story he regrets. A team of three wanted to finish an Elasticsearch cutover before Christmas Eve, so they worked “overlapping twelve-hour shifts for several days.” By the end, they were “Loopy, exhausted, and proud of ourselves.”

Someone had pasted an XML-escaped & straight into a JSON config. For months, “we were tokenizing on a.” It took a long time for anyone to notice.

He shares a few more examples. One Super Bowl weekend vanished while merging two modules that depended on different major versions of an HTTP library. Another case involved a velocity bonus, a rushed feature copied into multiple places, 45 to 60 minute deploys, and a bug caused by “a > that should have been a >=”.

Allen defines going slow in practical terms. “It means writing a planning document and coming to agreement on what we’re doing before anyone touches code.” “It means knowing how we’ll test the thing before we start building it.” Finally, “It means stopping when production is on fire at 2 a.m.” just to figure out what you actually know.

His team at Wallaroo worked this way deliberately. First, they created “a slow, throwaway ‘Wallaroo’ in Python” and built their test harness against that prototype. They also added fault injection “before we built the features that would need it.” When they picked Pony as their language, they willingly took on “Weeks of ramp time.” He sums it up simply: “We bought speed later by spending time up front.”

A rested engineer spots the & mistake, while a tired one ships it.

If you run agents: agents never tire, but the people approving their work do. A pace set by agents produces the twelve-hour shift without anyone deciding to have one.

Most design time is agreeing what words mean

One question and a single word started the & bug. A coworker asked, “We want to keep the same tokenization rules, right?” Allen replied: “I said yes.” Both had different rules in mind.

Distributed systems have a well known case of this problem. Tyler Treat wrote “You Cannot Have Exactly-Once Delivery.” back in 2015. He argued that the Two Generals problem makes it impossible over a network. He also noted how engineers “often bend the meaning of ‘delivery’ in order to make their system fit.”

Confluent announced exactly-once semantics for Kafka 0.11 in 2017. Their post answered him directly in the title: “Exactly-Once Semantics Are Possible.” They even quoted the skeptics: “Exactly-once delivery is impossible.” Kafka combined transactions with an idempotent producer to build it. For 1 KB messages, they measured only a 3% drop in throughput compared to ordered at-least-once delivery. Both sides were right. They disagreed about the word, not the maths. Treat was talking about network delivery. Confluent was talking about processing results within Kafka, relying on the exact idempotence escape hatch Treat had pointed out.

In 2017, researchers at Microsoft gave a similar trap a name: gray failure. Big cloud outages “tend to be caused by subtle underlying faults,” where one part struggles while another part misses the issue entirely. Their term for this is “differential observability.” Adding redundancy can actually make things worse. The system “works” under one definition while failing under another.

Released in September 2025, GitHub’s Spec Kit lays out the solution for agents: “Specs become the shared source of truth.” In this view, a spec “is a contract for how your code should behave.”

If you run agents: an agent will say yes to an ambiguous request faster than any colleague. Define the words in the spec before the agent starts: “delivered”, “done”, “same rules”.

Agents make upfront design pay off again

We stopped doing big upfront design because building took too long and plans went stale. Agents change that cost. When implementation takes a few hours, the plan becomes most of the work.

The major tool makers agree. GitHub’s Spec Kit points out that agent code often “looks right, but doesn’t quite work,” so it uses four phases: Specify, Plan, Tasks and Implement, where “you don’t move to the next one until the current task is fully validated.” AWS launched Kiro in July 2025. They built the whole product around spec-driven development because with vibe-coded prototypes “you can’t tell if the application meets them.”

In February 2026, OpenAI wrote a post on “harness engineering” about a product with roughly 1 million lines of code, around 1,500 pull requests, and “0 lines of manually-written code,” built by three engineers merging 3.5 pull requests each per day. “Plans are treated as first-class artifacts.” They enforce rules “mechanically via custom linters” alongside structural tests. Even with those checks, the team “used to spend every Friday (20% of the week)” cleaning up bad patterns the agents copied.

Anthropic’s Claude Code guidance is blunt about skipping verification. “Claude stops when the work looks done.” If it lacks a check it can run itself, “that makes you the test suite.” Anthropic suggests writing a spec and a definition of done, and using hooks for actions that must happen every time, since Claude “is allowed to decide a guideline doesn’t apply.”

You cannot trust your gut on speed either. A July 2025 METR trial found experienced developers took 19% longer using AI tools, even though they thought they were 20% faster. A follow-up in February 2026 hinted at a speedup, though METR calls this “only very weak evidence.”

Allen’s advice from an older era holds up well: put in the time early, and know how you will verify that the system works.

If you run agents: write the invariants, the spec and the test plan first. Hand the agent the plan, not the problem.

Before you hand the work to agents

Allen’s work comes down to one order of operations. Make the bug impossible. Test what is left with workloads under faults. Watch production, because something will get through. For packages and hardware, know what you trusted.

As checks a team can run:

1. Write invariants first. For each service, list what must never happen and what must eventually happen.

2. Push constraints into types. Sum types, exhaustive matches, validated constructors.

3. Prefer memory-safe languages for new code. Android’s data is the evidence.

4. Ask agents for properties, not examples.

Then the next four:

1. Run one simple workload under faults. A bank test for anything that moves value.

2. Gate on continuous checks, not approvals.

3. Cooldown dependencies in config, with a reviewed fast path for urgent fixes.

4. Give agents narrow capabilities and no long-lived secrets.

None of these needs a new tool. Most need a decision written down before code exists.

What to ask your team

• The risk: agent pull requests carry about 1.7 times more issues, and AI adoption still lowers delivery stability, per DORA 2025. Meanwhile packages ship worms through valid release pipelines.

• The cost: memory-safe code needed about 25% less review time at Google, and a 7-day cooldown would have blocked 8 of 10 recent attacks. Both are settings and choices, not headcount.

• Which invariants does each of our critical services have written down, and what checks them on every change?

• Do our agents install packages with a cooldown and without long-lived credentials?

• If we removed human approval tomorrow, which automated check would catch a lost write?

Reply to this email with one number from your team: the share of merged pull requests last month that were mostly written by an agent, and whether a human read every line. We will publish what readers send, without names, in a future edition.

Sources

• GitHub, “Octoverse: A new developer joins GitHub every second as AI leads TypeScript to #1”, 28 Oct 2025 (github.blog)

• CodeRabbit, “Our new report: AI code creates 1.7x more problems”, 17 Dec 2025 (coderabbit.ai)

• Google Cloud, “Announcing the 2025 DORA Report”, 23 Sep 2025 (cloud.google.com)

• DORA, “2025 State of AI-assisted Software Development” (services.google.com)

• Sean T. Allen, “Trunk Based Development: Now More Than Ever”, 21 May 2026 (seantallen.com)

• Sean T. Allen, “How Do I Know It’s Working?”, 27 Apr 2026 (seantallen.com)

• Sean T. Allen, “Spooky Action at a Distance”, 18 May 2026 (seantallen.com)

• Sean T. Allen, “Go Slow to Go Fast”, 7 May 2026 (seantallen.com)

• Sean T. Allen, “Teaching Claude to Write Pony”, 27 Mar 2026 (seantallen.com)

• Sean T. Allen, “Learning From Ada”, 22 Feb 2016 (seantallen.com)

• Sean T. Allen, “Of Debt and Decisions”, 19 Apr 2015 (seantallen.com)

• Sean T. Allen, “Will Wilson and Me”, 27 May 2026 (seantallen.com)

• Sean T. Allen, “On When Concurrency Matters”, talk, 21 Jul 2026 (seantallen.com)

• Sean T. Allen, “GC Made Fast”, talk and slides, 12 Apr 2026 (seantallen.com)

• Sean T. Allen, “Parma for Mortals”, 6 Mar 2026 (seantallen.com)

• Sean T. Allen, “On Deny Capabilities for Safe, Fast Actors”, Papers We Love, 8 Jan 2020 (seantallen.com)

• CoRecursive, “Unproven Tech Case Study with Sean Allen”, 2020 (corecursive.com)

• Antithesis, Bug Bash S2E8, “Why simple workloads find the hardest bugs”, 8 Apr 2026 (antithesis.com)

• Antithesis, “Depth is all you need: how Antithesis crushes Gradius”, 21 Feb 2025 (antithesis.com)

• Antithesis, “Why we taught agents to break distributed safety properties”, 25 Sep 2026 (antithesis.com)

• Antithesis, “Testing Datadog’s Next-Generation Event Platform Intake with Antithesis”, 29 Sep 2026 (antithesis.com)

• Antithesis, “Your agent can now ship reliable distributed systems”, 15 Oct 2026 (antithesis.com)

• AdaCore, “Should I choose Ada, SPARK, or Rust over C/C++?”, 27 Sep 2024 (adacore.com)

• Google Security Blog, “Eliminating Memory Safety Vulnerabilities at the Source”, 25 Sep 2024 (security.googleblog.com)

• Google, “Rust in Android: move fast and fix things”, 13 Nov 2025 (blog.google)

• CISA, “New Guidance Released for Reducing Memory-Related Vulnerabilities”, 24 Jun 2025 (cisa.gov)

• CISA and FBI, “Product Security Bad Practices”, v2.0, 17 Jan 2025 (cisa.gov)

• The Rust Reference, “Influences” (doc.rust-lang.org)

• The Rust Programming Language, “The match Control Flow Construct” (doc.rust-lang.org)

• Real World OCaml, “Lists and Patterns” (dev.realworldocaml.org)

• GitHub, “Why AI is pushing developers toward typed languages”, 8 Jan 2026 (github.blog)

• GitHub, “TypeScript’s rise in the AI era: Insights from Lead Architect, Anders Hejlsberg”, 6 Nov 2025 (github.blog)

• Mündler et al., “Type-Constrained Code Generation with Language Models”, Apr 2025 (arxiv.org)

• Yaron Minsky, “Effective ML Revisited”, 9 Mar 2011 (blog.janestreet.com)

• Martin Fowler, “Test Driven Development”, 11 Dec 2023 (martinfowler.com)

• Scott Wlaschin, “The Enterprise Developer from Hell”, 1 Dec 2014 (fsharpforfunandprofit.com)

• Claessen and Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs”, ICFP 2000 (cs.tufts.edu)

• Hypothesis documentation (hypothesis.readthedocs.io)

• Maaz et al., “Agentic Property-Based Testing”, 10 Oct 2025 (arxiv.org)

• Jepsen, “Analyses” (jepsen.io)

• Jepsen, “YugaByte DB 1.1.9”, 26 Mar 2019 (jepsen.io)

• Jepsen, “TiDB 2.1.7”, 12 Jun 2019 (jepsen.io)

• Jepsen, “CockroachDB beta-20160829”, 16 Feb 2017 (jepsen.io)

• Jepsen, “MariaDB Galera Cluster 12.1.2”, 16 Mar 2026 (jepsen.io)

• Jepsen, “TigerBeetle 0.16.11”, 6 Jun 2025 (jepsen.io)

• Kyle Kingsbury, “Jepsen: On the perils of network partitions”, 18 May 2013 (aphyr.com)

• Zhou et al., “FoundationDB: A Distributed Unbundled Transactional Key Value Store”, SIGMOD 2021 (foundationdb.org)

• Will Wilson, “Testing Distributed Systems w/ Deterministic Simulation”, Strange Loop, 20 Sep 2014 (youtube.com)

• TigerBeetle, “October in TigerLand”, 7 Nov 2024 (tigerbeetle.com)

• AWS, “Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region”, Oct 2025 (aws.amazon.com)

• Lorin Hochstein, “Quick thoughts on the recent AWS outage”, 25 Oct 2025 (surfingcomplexity.blog)

• Lorin Hochstein, “AWS re:Invent talk on their Oct ‘25 incident”, 14 Dec 2025 (surfingcomplexity.blog)

• James Reason, “Human error: models and management”, BMJ, 18 Mar 2000 (ncbi.nlm.nih.gov)

• DORA, “Accelerate State of DevOps Report 2024” (services.google.com)

• Octopus Deploy, “2024 DevOps performance clusters”, 19 Nov 2024 (octopus.com)

• Newcombe et al., “Use of Formal Methods at Amazon Web Services”, 29 Sep 2014 (lamport.azurewebsites.net)

• Sadowski et al., “Lessons from Building Static Analysis Tools at Google”, CACM 2018 (research.google)

• Pony tutorial, introduction, “Object Capabilities”, “C FFI”, “Trust Boundary” and “Garbage Collection” (tutorial.ponylang.io)

• Charity Majors, “I test in prod”, Increment, 23 Aug 2019 (increment.com)

• Ward Cunningham, “The WyCash Portfolio Management System”, OOPSLA, 26 Mar 1992 (c2.com)

• Martin Fowler, “Technical Debt”, 21 May 2019 (martinfowler.com)

• Joel Spolsky, “Things You Should Never Do, Part I”, 6 Apr 2000 (joelonsoftware.com)

• CISA, “Supply Chain Compromise of Third-Party tj-actions/changed-files”, 18 Mar 2025 (cisa.gov)

• Nx, security advisory GHSA-cxm3-wv7p-598c, 27 Aug 2025 (github.com)

• Wiz, “s1ngularity: supply chain attack leaks secrets on GitHub”, Aug 2025 (wiz.io)

• Aikido, “npm debug and chalk packages compromised”, 8 Sep 2025 (aikido.dev)

• CISA, “Widespread Supply Chain Compromise Impacting npm Ecosystem”, 23 Sep 2025 (cisa.gov)

• Check Point Research, “Shai-Hulud 2.0: Inside The Second Coming”, 25 Nov 2025 (blog.checkpoint.com)

• GitHub, “Our plan for a more secure npm supply chain”, 22 Sep 2025 (github.blog)

• GitHub Changelog, “npm classic tokens revoked, session-based auth and CLI token management now available”, 9 Dec 2025 (github.blog)

• Aqua Security, Trivy advisory GHSA-69fq-xp46-6x23, 19 Mar 2026 (github.com)

• LiteLLM, “Security Update: Suspected Supply Chain Incident”, Mar 2026 (docs.litellm.ai)

• Datadog Security Labs, “axios npm supply chain compromise”, 31 Mar 2026 (securitylabs.datadoghq.com)

• StepSecurity, “Mini Shai-Hulud is back”, 11 May 2026 (stepsecurity.io)

• Chainguard, “Mini Shai-Hulud npm Attack: AntV Ecosystem Compromise”, 19 May 2026 (chainguard.dev)

• Wiz, “keyv and cacheable npm Package Hijacked in Supply Chain Attack”, 4 Aug 2026 (wiz.io)

• Adnan Khan, “Clinejection”, 9 Feb 2026 (adnanthekhan.com)

• Spracklen et al., “We Have a Package for You!”, USENIX Security 2025 (arxiv.org)

• Cloud Security Alliance, “Slopsquatting: AI Code Hallucinations Fuel Supply Chain Attacks”, 19 Apr 2026 (labs.cloudsecurityalliance.org)

• William Woodruff, “We should all be using dependency cooldowns”, 21 Nov 2025 (blog.yossarian.net)

• Andrew Nesbitt, “Package managers need to cool down”, 4 Mar 2026 (nesbitt.io)

• Cal Paterson, “Dependency cooldowns turn you into a free-rider”, Apr 2026 (calpaterson.com)

• CPAN mirror, Funet (nic.funet.fi)

• npm Docs, “scripts” (docs.npmjs.com)

• Socket, “pnpm 10.0.0 blocks lifecycle scripts by default”, 10 Jan 2025 (socket.dev)

• Aleksa Sarai, runc security advisory, oss-security, 5 Nov 2025 (openwall.com)

• Mark S. Miller, “Robust Composition”, PhD dissertation, May 2006 (erights.org)

• Clebsch et al., “Deny Capabilities for Safe, Fast Actors”, 2015 (ponylang.io)

• Deno docs, “Security and permissions” (docs.deno.com)

• WASI, “Capabilities” (github.com)

• Google Project Zero, “Reading privileged memory with a side-channel”, 3 Jan 2018 (projectzero.google)

• Graz University of Technology et al., Meltdown and Spectre (meltdownattack.com)

• Brendan Gregg, “KPTI/KAISER Meltdown Initial Performance Regressions”, 9 Feb 2018 (brendangregg.com)

• Johnson et al., “Parma: Confidential Containers via Attested Execution Policies”, Feb 2023 (arxiv.org)

• TEE.fail, IEEE S&P 2026; disclosed Oct 2025 (tee.fail)

• Battering RAM, IEEE S&P 2026; disclosed Oct 2025 (batteringram.eu)

• Discord, “Why Discord is switching from Go to Rust”, 4 Feb 2020 (discord.com)

• Rick Hudson, “Getting to Go: The Journey of Go’s Garbage Collector”, 12 Jul 2018 (go.dev)

• OpenJDK, JEP 333, JEP 376 and JEP 439 (openjdk.org)

• Clebsch et al., “Orca: GC and Type System Co-Design for Actor Languages”, OOPSLA 2017 (ponylang.io)

• Martin Fowler, “The LMAX Architecture”, 12 Jul 2011 (martinfowler.com)

• Jeff Bonwick, “The Slab Allocator”, USENIX Summer 1994 (people.eecs.berkeley.edu)

• Go issue 51317, “proposal: arena: new package providing memory arenas”, 22 Feb 2022 (github.com)

• Tyler Treat, “You Cannot Have Exactly-Once Delivery”, 25 Mar 2015 (bravenewgeek.com)

• Confluent, “Exactly-Once Semantics Are Possible: Here’s How Kafka Does It”, 30 Jun 2017 (confluent.io)

• Huang et al., “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems”, HotOS 2017 (microsoft.com)

• GitHub, “Spec-driven development with AI: Get started with a new open source toolkit”, 2 Sep 2025 (github.blog)

• Kiro, “Introducing Kiro”, 14 Jul 2025 (kiro.dev)

• OpenAI, “Harness engineering: leveraging Codex in an agent-first world”, 11 Feb 2026 (openai.com)

• Anthropic, “Best practices for Claude Code” (code.claude.com)

• METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 10 Jul 2025 (metr.org)

• METR, “Uplift update”, 24 Feb 2026 (metr.org)

Each model replied once per game, so Opus’s lead over Astra is unproven. Opus also wrote our code while being one of the models compared, which is why we ran MIT’s own networks and answers through it.

The game footage and network activations come from the saved network, replayed for this video. Published scores are the mean of 1,000 games.

Run the checks yourself: https://github.com/bolnet/context-clock/blob/master/WEIGHTSPACE_HOW_WE_DID_IT.md

Sources:

MIT CSAIL, Sparks of Weight-Space Programming (Takehi, Teoh, Zhang, Khattab): https://rikiyat.github.io/blog/weight-space-programming/

MIT’s logs: https://github.com/RikiyaT/lm_understand-logs

Our runs and checker: https://github.com/bolnet/context-clock

If an AI can now repair another AI, who should be allowed to make those repairs, and how would you know they did?

Sources: Takehi, Teoh, Zhang and Khattab, “Sparks of Weight-Space Programming”, MIT CSAIL, 2 Nov 2026 (rikiyat.github.io). Matthew Honnibal, critique, 3 Nov 2026 (honnibal.dev). Qwen3.5-0.8B model card (huggingface.co).

In 60 seconds

• One engineer’s merged pull requests went from 19 a month to 207 when coding agents arrived.

• Across GitHub, agents started 17% of pull requests in March 2026 and more than 40% by September. Per developer, the gain was only 16.2% to 20%.

• The gain arrives when teams remove human steps. 96% of his PRs at Tano never had a human reviewer, and median time to merge fell from 15.1 hours to 0.7.

• Each removed step exposes the next. At Anthropic, CI jobs rose 25 times in six months, then merges went from 2,500 a day to more than 30,000 and Anthropic left git.

• The last constraint is the engineer. Agent sessions sit about 80% idle waiting for a person, and follow-up fixes nearly doubled, from about 7.6% to about 13.8%.

One engineer’s merged pull requests went from 19 a month to 207

Neil Kakkar publishes his own GitHub ledger. It covers three jobs. He was at Bloomberg from 2018 to 2021. He was at PostHog from May 2021 to January 2025. He has been the founding engineer at Tano since January 2026. Tano builds agents that run influencer marketing for brands.

By his own count, he merged 8 pull requests a month at Bloomberg and 19 at PostHog. At Tano the figure is 207 per 30 days. That is 859 merged PRs in 3.7 years at PostHog, against 1,638 in the first eight months at Tano. His Tano total passed his whole PostHog total on day 139.

The jump from Bloomberg to PostHog was 2.4x. He credits that to a faster company. The jump from PostHog to Tano is 11x, and it came with coding agents. He also gives a smaller number. Comparing his best eight months at PostHog with his first eight at Tano gives 7.0x. He calls that his most conservative figure. We use both.

His peak months tell the same story. PostHog’s best month had 42 merged PRs. Tano’s best had 363. On his busiest Tano days, 44 PRs merged. Lines added per 30 days went from 2,899 to 102,239. No one types 100,000 lines a month by hand.

Two caveats come from the ledger itself. He labels PostHog “pre-agent” and says the label is approximate. Copilot and chat assistants existed for part of his time there. Autonomous agents did not. He also says a PR count measures throughput, not value. A pull request is a unit of change, not a unit of worth.

So treat 11x as an upper bound for one person in one kind of company. It is still the cleanest before-and-after we have. It is one engineer, measured the same way, across a clear change in tools and process. Most public data mixes thousands of teams with different habits. This ledger does not.

The rest of this edition asks what had to change around him for that number to happen, and what it cost.

Across GitHub, agents now start four in ten pull requests

One ledger is a story. The industry numbers give the scale.

GitHub’s Octoverse 2026, published October 28, counted 1.3 billion merged pull requests in its year. GitHub says that more than tripled. The count now includes private repositories, so it does not line up neatly with last year’s report. The share of PRs started by an agent is the cleaner signal. It was 17% in March 2026. By September it was more than 40%. GitHub says activity climbed from January and surged around June. AI code review usage rose 90% year on year. Actions now runs 1,440 jobs a second.

The Stack Overflow 2026 survey, published October 6, asked more than 30,000 developers. Among those who use AI, 66% use coding agents. Claude Code was used by 66% of them and Copilot by 59%. A third of daily users spend four or more hours a day in AI tools.

Per developer, the gain is much smaller than 11x. Faros AI studied 22,000 developers across more than 4,000 teams for its April 2026 report. The PR merge rate per developer rose 16.2%. Cortex’s 2026 benchmark found PRs per author up 20%.

So the totals tripled while the typical developer merged about a sixth to a fifth more. Both can be true. More developers, more agents and more automation add volume. The individual gain depends on what surrounds the individual.

That gap, between 11x for one engineer and about 20% for the median, is the subject of this edition. The difference is not the model. Neil and the Faros teams had access to the same agents. The difference is which human steps were still in the path between a finished change and production.

Review was the first wall, and it fell first

Code review was the step most teams kept. It was also the first to break under agent volume.

Cloudflare put it plainly in April 2026. Review is “one of the most reliable ways to bottleneck an engineering team.” Boris Cherny, who leads Claude Code at Anthropic, gave the method in June: “Find the bottleneck, solve the bottleneck.”

Faros measured what happens when review stays manual. Median time in review rose 441.5%. Pull requests merged with no review at all rose 31.3%. Volume went up, so people either queued or skipped.

Neil’s ledger shows the other path. At PostHog he reviewed 1,589 of his teammates’ pull requests. At Tano he has reviewed 67. Per merged PR, that ratio went from 1.85 to 0.04. At Bloomberg it was about 2. By his count, 96% of his Tano PRs never had a human reviewer. A bot, Claude, reviewed 98% of them. His page says engineers at Tano “own their areas end to end.”

The effect on speed is direct. His median time from opening a PR to merging it went from 15.1 hours at PostHog to 0.7 hours at Tano. The share merged within an hour went from 23% to 57%.

Picture an engineer with today’s agents dropped into a team where every PR needs a colleague’s approval. Output rises, but only until the queue fills. Every PR adds work for someone else. The faster you write, the longer the line. Faros calls review the new bottleneck for exactly this reason.

So the first removed constraint was not typing. It was waiting for another person’s attention. Teams that kept mandatory human review saw small per-person gains. Teams that replaced it saw large ones. The next question is what replaced it, and whether it holds up.

Big companies now let AI carry most review

Replacing human review is no longer a startup habit. Several large companies now publish their numbers.

Cloudflare ran AI review on 48,095 merge requests across 5,169 repositories in 30 days. That was 131,246 review runs. The median run took 3 minutes 39 seconds and cost $0.98. The average was $1.19. Engineers used a break-glass override on 0.6% of merge requests. Cloudflare sorts every merge request into one of three risk tiers and reviews each tier differently.

Intercom wrote in April 2026 that over 93% of its pull requests are agent-driven. Over 19% are “auto-approved with no human reviewer.” In four weeks, 497 PRs went fully autonomous. The backend revert rate was 0.53% for AI-authored code and 5.39% for human-authored code. Downtime fell 35% while deployments doubled.

Anthropic launched Code Review for Claude Code in March. Inside Anthropic, the share of PRs with substantive review comments went from 16% to 54%. Engineers marked under 1% of findings as incorrect. Each review costs $15 to $25. Cherny says a team of Claude reviewers catches “pretty much every bug,” and a human still approves.

Google reported the scale in April. Sundar Pichai said 75% of all new code at Google is now AI-generated and approved by engineers.

The building blocks are documented. Claude Code GitHub Actions run a review on every pull request event and post inline comments. A deeper multi-agent review, called ultrareview, costs $5 to $25 and runs for 5 to 15 minutes. Claude does not start it on its own. A person asks for it on a risky change.

The pattern across these companies is the same. A cheap reviewer reads everything. A deeper reviewer runs on demand. Humans stay for the riskiest tier. Nobody here removed review. They moved it to a machine and kept humans at the top of the risk ladder.

Agents miss what the codebase already knows: money

An AI reviewer reads the diff. It does not always know what the rest of the codebase already handles. Money is the classic trap.

Many developers assume every currency has 100 minor units, like cents in a dollar. Many do not. The official ISO 4217 list, maintained by SIX and last published September 17, 2026, gives the minor unit for each currency code. By our count of that list, 139 codes use 2 decimals. 17 use none, including the Japanese yen and the Korean won. 7 use three, including the Bahraini dinar and the Kuwaiti dinar. 2 use four. That is 26 currencies where “multiply by 100” is wrong.

Stripe’s documentation states the rule that payment APIs follow. Amounts are given “in the currency’s minor unit.” Stripe documents zero-decimal and three-decimal currencies separately for this reason.

Agents can parse a currency amount as a float and multiply by 100. That passes every test that uses dollars. It breaks in Bahrain and in Japan. The bug ships when the codebase already has a currency utility and the agent writes its own instead. A human who knows the codebase would point at the utility. A reviewer that reads only the diff may not.

There are three fixes, and they stack. Store money as integers in minor units, never as floats. Keep one currency module and make the review agent check that new code uses it. Add test cases for a zero-decimal and a three-decimal currency, so the yen and the dinar break the build before production does.

The general point is about context. A review agent is only as good as what it can see. Give it repository search. Give it a rules file that names the shared modules. Otherwise it reviews code in isolation, and the bugs that slip through are the ones that only make sense in context.

A normal index build on a big table blocks writes

The second trap is the database. Agents write migrations quickly. Some migrations are safe on a small table and dangerous on a large one.

PostgreSQL’s documentation is direct. A standard index build “locks out writes (but not reads)” on the table until it finishes. On a table with millions of rows, that can take minutes. Every insert and update waits. If your ingestion pipeline writes to that table, ingestion stops. If deploys run migrations, deploys stall behind the lock.

The fix is in the same documentation. CREATE INDEX CONCURRENTLY builds the index without blocking writes. It has its own rules. It cannot run inside a transaction block. If it fails, it leaves an “invalid” index behind that must be dropped and rebuilt.

This is old knowledge, and teams already encode it. The strong_migrations library for Rails says it plainly: “In Postgres, adding an index non-concurrently blocks writes.” GitLab’s database guide makes add_concurrent_index a requirement on populated tables. A RuboCop rule, PreventIndexCreation, blocks new indexes on tables GitLab marks as large. Squawk, a Postgres migration linter, has a rule to “Ensure all index creations use the CONCURRENTLY option.”

An agent writing a migration does not know which of your tables are large unless you tell it. A human reviewer who has been paged for this before would catch it. Remove that reviewer, and the knowledge has to live somewhere the agent can read.

The cheapest place is the review agent’s rules. List your largest tables. Forbid blocking migrations on them. Require concurrent index builds. Then add a linter like Squawk to CI, so the rule holds even when the review agent misses it. Two layers cost little. One outage on a large table costs far more.

Every incident becomes a rule the reviewer enforces

The currency bug and the index lock share a shape. Each is a mistake a human reviewer learned to catch once, through pain. Without that reviewer, the lesson has to be written down.

The tools now support this directly. Claude Code’s code review reads a REVIEW.md file in the repository. The documentation says to “add rules you want flagged on every pull request.” Each rule is a sentence the reviewer applies to every change.

Every, a company that builds several AI products, describes the habit it calls compound engineering. When something goes wrong, “add a note so the agent learns.” Each bug, failed test or insight “gets documented and used by future agents.” Every runs 12 review subagents in parallel on its code. In its words, bug fixes “eliminate entire categories of future bugs.”

GitLab’s PreventIndexCreation rule is the same idea from before agents. A past incident became a lint rule that blocks the whole class of change.

A rule that has never been tested is a guess. We would run each new rule through a short loop before trusting it.

First, trace the incident to the pull request that caused it. Second, write the rule as one sentence in the rules file. Third, replay that pull request through the reviewer and confirm the rule flags it. Fourth, run the reviewer over the last 50 merged PRs and count false flags. A rule that flags half the codebase will be ignored. Fifth, version the rules file and enforce it on every PR.

This turns a team’s scar tissue into something an agent can read. Done well, the same bug does not ship twice. Done badly, the rules file grows into noise that nobody trusts. The replay step is what separates the two.

Reverts fell. Follow-up fixes nearly doubled

The quality picture is mixed, and Neil’s ledger shows why one metric misleads.

By his count, the share of his PRs that were later reverted fell from 2.3% at PostHog to 0.4% at Tano. That looks like 6x fewer mistakes. His own page warns against reading it that way. Reverts undercount rework. At Tano, he writes, “I fix forward instead of reverting.”

So he tracks a second number. The share of his PRs that needed a follow-up fix within seven days went from about 7.6% at PostHog to about 13.8% at Tano. Fewer reverts, nearly twice the follow-up fixes. His PRs also grew. Median lines added went from 42 to 229. Median files touched went from 3 to 6. About 30% of his Tano PRs fix bugs.

The wider data points the same way. Faros found bugs per developer up 54% and the ratio of incidents to PRs up 242.7%. Cortex found incidents per PR up 23.5% and change failure rate up 30%.

Intercom’s numbers show it can go the other way. Its backend revert rate was 0.53% for AI-authored code against 5.39% for human-authored code. The difference is in the guardrails around the code, not only in who wrote it.

Fixing forward is a choice, and it is not the default advice. Google’s SRE workbook names rollback as the generic mitigation that “speeds recovery.” DORA’s metrics count a rollback and a hotfix the same way in the change fail rate. DORA also tracks deployment rework rate, the unplanned deploys caused by incidents.

The practical rule is to track three numbers together: volume, size and rework. Any one alone tells a happy story. A falling revert rate with rising follow-up fixes means the cost moved, not that it disappeared.

Fast rewrites turn big migrations into two-way doors

Speed changes which decisions are reversible.

In Amazon’s 2015 shareholder letter, Jeff Bezos split decisions into two kinds. One-way doors are hard to undo and deserve care. Most decisions, he wrote, are “changeable, reversible” two-way doors. Those should be made “quickly by high judgment individuals or small groups.”

Large data migrations used to be one-way doors. PostHog’s persons-on-events migration is a public example. PostHog moved person data onto its events table so that queries “no longer have to join tables.” PostHog said queries would improve “by up to 400%.” It had tracked over 50 billion events by then. Neil’s page dates the work from May 2022 to May 2023, about a year. He contributed to one company-wide migration at PostHog and led one.

At Tano, by his count, he has led four company-wide migrations in eight months. One moved Tano’s creator agent from a checklist design to workflow orchestration. Another moved 13,345 partners to a new model between August and September 2026.

His June post describes what changed. “A big overhaul doesn’t take weeks. It can be done overnight.” When a rewrite is that cheap, architecture becomes a two-way door. You can try a design in production, learn, and change it.

That has a cost. Moving fast through migrations means some break. A team that treats every migration as reversible needs the guardrails from the last three sections: rules for dangerous operations, tests for edge cases, and a way to measure rework.

It also changes planning. When a migration took a year, teams future-proofed it, because they could not afford a second one. When it takes weeks, the better plan is often the smallest design that works now, with the expectation of changing it. The risk is in pretending a one-way door is a two-way door. Data loss, payments and security changes still deserve the slow path.

From a checklist agent to a workflow with fewer tools

Neil’s first Tano migration is a design lesson for anyone running agents in production.

Tano’s agents work with creators for brands. Seedcamp described the company in November 2025 as an end-to-end AI marketing business whose agents “source, vet, outreach, negotiate.” Neil’s page lists a negotiation engine behind more than $1 million in accepted creator deals.

His page also lists a project called “Checklists to workflow orchestration,” from February to July 2026. The workflow engine now runs more than 97,000 collaborations. One example PR in his ledger replaces a “v1 one-shot executor” with an “agentic graph invoker.”

Anthropic’s guide to building effective agents draws the same line. Workflows follow “predefined code paths.” Agents “dynamically direct their own processes and tool usage.” The guide recommends the simplest design that works, and adding agent freedom only where it pays.

The failure mode of the free design is common. One agent gets a long checklist and a large set of tools. It picks the wrong tool at the wrong step. When it goes wrong, nobody can see why, because the steps are not separate.

The workflow design splits the job. Each task is a step with a clear input, a clear output and a short list of allowed tools. A failed step shows up in the trace. A wrong tool call is impossible if the tool is not offered.

That fits the rest of this edition. Agents remove human constraints. Production systems need new ones, built into the design. A workflow with narrow tools is a constraint, chosen on purpose. It trades some flexibility for something you can observe and fix.

If you run an agent with more than a dozen tools and a free-form plan, look at your traces. If the same wrong tool call keeps showing up, the fix is probably structure, not a better prompt.

One person can now own a whole feature

Review was one human constraint. Coordination was another. When writing code was slow, splitting a feature across several people made sense. When it is fast, the hand-offs cost more than the code.

Every runs five products. Each is “primarily built and run by a single person.” Neil’s page says the same of Tano: engineers “own their areas end to end.” In his March post on working with Claude Code, he describes his own role as “the manager of agents,” not the implementer. By his count he built 24 projects in eight months at Tano, against 17 in 3.7 years at PostHog.

Anthropic surveyed 132 of its own engineers for a December 2025 study. Claude was used in about 60% of their work. Engineers said they were “becoming more ‘full-stack’.” One said he is now “70%+” a reviewer and reviser of code.

OpenAI published a related case in February. A small team shipped a beta of about a million lines over five months with no hand-written code.

PostHog shows that single ownership predates agents. Its handbook says engineers are “normally responsible for their area of the product.” It expects product engineers to talk to users, and it built programs to “reduce the need for PMs.” PostHog had about 25 people when Neil joined in 2021.

What changes with agents is the size of the area one person can own. Design, frontend, backend and data work all fit in one head when agents write most of the code.

The cost moves to choosing. Neil wrote in February that “choosing what to build has become more important than just building.” A team of single owners needs a clear shared priority list, or each owner optimizes their own corner. The planning meeting matters more, not less. So does writing down why each area works the way it does, which is where the next part of this edition begins.

The bottleneck moved into the engineer’s head

Remove review and coordination, and the work still has to pass through one place: the person deciding what the agents do.

Anthropic said it directly in its October 16 report on employment at Anthropic. Human attention is now “the bottleneck for agent output.” Agent sessions spend about 80% of their time idle, waiting for a person. One engineer described “an army of Claudes, with every one asking for my attention.”

Simon Willison has written about this more than anyone. In February he said “the productivity boost these things can provide is exhausting.” In April he said using agents well takes “every inch of my 25 years” and is “mentally exhausting.”

Steve Yegge calls it the AI Vampire. His February essay says AI leaves “difficult decision-making for humans, which is exhausting and draining.” On the Hanselminutes podcast he said working with AI “drains human energy,” and asked: “so why is everyone so exhausted?”

The survey data agrees. Harness surveyed 700 engineering staff in February 2026. 75% said pressure to ship fast contributed to burnout.

From the inside it looks different. The ledger shows more output, and the engineer feels the cost. Agents do the typing. The person does the deciding, the checking and the switching, all day, at a pace set by machines that never tire.

That cost shows up in four places. The engineer switches between too many threads. Shared context thins out when nobody reads other people’s code. The workday stretches. And the bill for tokens sets a new kind of limit. The next sections take them one at a time, with what the leading practitioners say they do about each.

Too many agents at once is thrashing

Operating systems have a word for this. In 1968 Peter Denning defined thrashing as “excessive overhead and severe performance degradation or collapse caused by too much paging.” The machine spends its time switching instead of working. The fix was to run fewer processes at once.

Engineers with agents now face the human version. Neil’s ledger shows it. In a typical month at PostHog he had about 2 projects in flight, at most 5. At Tano he has about 6, at most 9. His March post describes going “from getting overwhelmed by two parallel branches to running five worktrees at once.”

His session logs show a more modest picture than the projects suggest. PRs in flight at the same time went from 1.85 to 2.9. Half his active agent time is a single session.

Addy Osmani of Google wrote the clearest line on it in April: “Your cognitive bandwidth doesn’t parallelize.” The cost, he says, “isn’t in the switch itself - it’s in the recovery time.” Agent decisions cannot queue up like email.

Willison describes the limit in practice. He can “fire up four agents in parallel,” and “by like 11 AM, I am wiped out.”

A Harvard Business Review study by BCG and UC Riverside, published in March, surveyed 1,488 US workers. Heavy oversight of AI came with 14% more mental effort, 12% more fatigue and 19% more information overload. Self-rated productivity fell once people used four or more AI tools.

The operating system fix still applies. Cap the number of agent sessions per person. Finish before starting. Let urgent bugs interrupt, but not every bug. Faros found 26% more in-progress tasks sitting idle for seven days or more. That is what thrashing looks like on a team board.

Nobody reads every pull request anymore

Code review did more than catch bugs. It spread knowledge. PostHog’s handbook says it plainly: “Reviews also build shared context and collective ownership.”

That benefit goes away with the review. Neil went from reviewing 1,589 teammates’ PRs at PostHog to 67 at Tano. He no longer sees most of what his colleagues change. Neither do they see his.

Anthropic’s own engineers report the same thing. In its December study, one said “More junior people don’t come to me with questions as often.” Another sends “80-90%” of questions to Claude instead of colleagues. A January Anthropic trial with 52 engineers found that people who learned a new library with AI help scored 50% on a comprehension quiz, against 67% for those without it.

Charity Majors put the stakes in one line in July: “The real product of a software team is shared understanding.”

The risk has a name. The bus factor, or truck factor, is the number of people who must leave before a project stalls. A 2016 study of 133 popular GitHub projects found that 65% had a truck factor of two or less. When one person owns an area end to end and nobody reads their code, the factor for that area is one.

The fix the industry is settling on is written context. AGENTS.md is a file that tells agents how a codebase works. Its site calls it “a README for agents” and says over 60,000 open-source projects use it. In a monorepo, “the closest AGENTS.md to the edited file wins,” so each area can carry its own. OpenAI’s Codex documentation gives the same advice.

Neil’s ledger shows the habit at work. His documentation lines at Tano reached 24,750, against 2,949 at PostHog. Written for agents, it also serves the next human.

The workday changed shape

When a person wrote all the code, the day had a rhythm. Write in the morning, open PRs in the afternoon, wait for approval.

Neil’s charts show that rhythm at PostHog. On a typical day in April 2023 he opened 2 PRs, both between about 13:00 and 14:45. At Tano, on a typical day in August 2026, he opened 10, spread from about 10:10 to 17:35. PRs now land when an agent finishes, not when he does. The share of his PRs opened before 11:00 rose from 11% to 19%, because agents keep working after he leaves.

That creates a new pressure. If the agents could always be working, every pause feels like waste. Willison described people “losing sleep” over the thought that “my coding agents could be doing work for me,” then waking “at four in the morning.”

The data shows the hours stretching. Harness found 71% of engineering staff work evenings or weekends at least weekly. A Berkeley study of 200 employees, reported in Harvard Business Review in February, found people worked faster, took on more tasks, and worked longer hours, with “constant task-switching.” Stack Overflow found a third of daily AI users spend four or more hours a day in AI tools.

The people furthest ahead set limits. Mitchell Hashimoto, who founded HashiCorp, advises you to “turn off agent desktop notifications” and decide yourself when to check in. Yegge’s advice is simpler: “close the laptop and go for a walk.”

Good pacing looks like this. Put the hardest decisions where your attention is best. Do not make risky changes when you are tired. Let agents run on clear, bounded tasks while you are away, and review them fresh. Set a hard end to the day. The agents do not need rest. You do, and your judgment is now the scarce input.

Default prompts make products look alike

Agents can build a user interface in minutes. The problem is that they build the same one.

Anthropic described this in November 2025. Its applied AI team called it “distributional convergence.” Without guidance, Claude will “almost always conform to Inter fonts, purple gradients on white backgrounds.” Users call the result “AI slop.” Anthropic’s own front-end design skill now warns against patterns that “read as AI-generated,” such as fade-and-slide-up entrances and the generic SaaS card layout.

The reason is in how models work. They produce the most likely output, and the most likely website is an average of many websites. Speed makes the average cheap. Distinctive design still takes taste and decisions.

PostHog is a useful contrast. Its first non-engineering hire was an artist, Lottie Coxon, who drew the company’s hedgehog mascot. Co-founder James Hawkins said, “We’re not going to win by being the same.” The brand that followed is one of the most recognizable in developer tools. That was a human choice, made early, before speed was cheap.

For AI-native teams the lesson is that design is now a constraint you have to add on purpose. A small team of single owners, each moving fast, will ship the default look unless someone owns the brand.

There are concrete fixes. Give agents a design system: tokens for color and type, a component library, and examples of your own pages. Anthropic’s skills approach does this, and its post shows the output change when the model is given explicit direction. Name the patterns to avoid, not only the ones to use. And review the first screen of every new product with the same care as its database migrations.

Plan limits, not ambition, now set the pace

The last new constraint is money, and it shows up as limits.

Claude’s plans set the shape of heavy agent use. As of November 3, 2026, Max costs $100 a month for “5x Pro usage every 5 hours and every week.” Ultra costs $200 a month for 10x weekly usage, with no five-hour limit. Anthropic’s support page says Pro and Max plans have “a five-hour session limit and a weekly limit.” Usage beyond that is billed “at standard API rates.”

API prices per million tokens are $4 in and $20 out for Opus 5.5, $2 and $10 for Sonnet 5.5, and $10 and $50 for Fable 5.1. An engineer running several agents all day can use far more than a plan covers.

The industry has a word for maximizing usage. Tomasz Tunguz defined “tokenmaxxing” in April as “the deliberate practice of maximizing token consumption,” and described about 250 million tokens in one day. IBM reported in September that some companies were “incentivizing employees to maximize their AI token usage,” including a leaderboard at Meta. The BCG study found firms sometimes use token consumption as a proxy for performance.

That is the wrong metric. Token spend measures effort, not results. Review shows the better frame. Cloudflare’s AI review averages $1.19 per merge request. Anthropic’s deeper Code Review costs $15 to $25. Cherny calls multi-agent review expensive, token-heavy computation. Those prices are worth paying for some changes and not others.

Autonomous monitoring, where agents watch production and open PRs on their own, is where limits bite next. PostHog’s Replay Vision, launched in September, watches session recordings “so nobody has to sit through them.” Sentry’s Seer can “open pull requests for us automatically,” but only for issues it judges fixable.

Measure cost per merged PR and cost per prevented incident. A limit is useful when it forces you to spend tokens on the work that matters most.

Anthropic sped up review, and CI broke next

The clearest record of a constraint moving comes from Anthropic itself.

In a September 14 post, Anthropic’s Sachin Malhotra described what happened after agents took over most code writing. Anthropic produces about 8 times more code per quarter than it did in 2021 to 2025. Claude writes about 80% of it. Tests grew 10 times while headcount stayed roughly flat. CI jobs rose 25 times in six months.

The post names the order. “Once PR review gets accelerated, CI starts feeling the pressure.”

The first thing to break was test selection. Anthropic ran only the tests affected by each change. A single process, called the listener, tracked which tests touched which code. It fell behind the PR queue. The first pages came in October 2025. Twenty minutes of listener lag meant tens of thousands of test updates were not applied.

The team patched it three times, and each patch lasted less time than the one before. A bigger machine bought 70 days. Sharding bought 29. Daily restarts bought less than one.

Then one engineer rebuilt it in three weeks. State moved to an in-memory store. Stateless workers append their results to a journal. The new design scales with the number of workers, not the size of one machine.

This is the pattern behind the 11x ledger at company scale. Remove the human step, and the next system in line takes the load. That system was designed for human pace. It holds for a while, with patches. Then the patches stop working and it needs a new design.

Teams adopting agents should expect this sequence and watch for it. If your CI queue time is rising faster than your PR count, CI is next. The time to redesign is when the second patch lasts half as long as the first, not after the third.

Then source control broke, and Anthropic left git

After CI, the next system was source control. Anthropic described it in an October 13 post.

Merged changes to its monorepo went from about 2,500 a day in May 2026 to more than 30,000 a day in September. The main history is about 100 GB. CI ran about 25 million checkouts a day. Fetched the naive way, that would have meant more than 600 petabytes of git traffic a day. At peak, Anthropic ran several hundred git mirrors to keep up.

Anthropic names the root cause. Git was designed to work offline. Its clients are developers who sync occasionally. Agents are always online and sync constantly. The design that served people for twenty years worked against machines.

So Anthropic built its own version control system, called Scion, and a code review system called Antique. A team of seven moved the monorepo off git in five months. The new system uses about 200 times less CPU and memory than the git mirrors. Checkouts went from minutes to seconds. CI gained 25% more CPU capacity. The cache hit rate is above 99%.

Most teams will never reach 30,000 merges a day. The lesson still applies. Each removed constraint exposes the next one. The order at Anthropic was review, then CI, then source control. Each system was built for human pace and broke under agent pace.

Google reports the same kind of change from the other direction. Pichai said an agent-assisted code migration finished “six times faster than was possible a year ago with engineers alone.”

The useful question for any team is which of your systems assumes a human is on the other end. Rate limits, queue sizes, review tools, deploy windows and on-call rotations all carry that assumption. List them before agents find them for you.

After the machines, the people became the constraint

Anthropic’s October 16 report on employment at Anthropic describes the step after CI and git. The constraint is now people.

Agent sessions spend about 80% of their time idle, waiting for human input. Human attention is “the bottleneck for agent output.” Work on the agents themselves, such as tooling, pipelines and prompts, rose from 10% of sessions in fall 2025 to 22% in spring 2026. The report also names the human side. An executive said “a workforce that is anxiously worried is not a workforce doing its best work.”

This fits a model older than software. Eliyahu Goldratt’s theory of constraints says a system moves at the pace of its tightest step. Improve that step, and another becomes the tightest. His last step is to avoid inertia and go back to the start. In agent terms: find the bottleneck, remove it, then look again.

Cherny’s line, “Find the bottleneck, solve the bottleneck,” is the same loop. It explains why the sequence keeps going. Review fell, so CI broke. CI was rebuilt, so git broke. Git was replaced, so attention is the limit.

Attention is harder to scale than a server. It cannot be sharded. That is why the costs in the middle of this edition matter. Thrashing, lost shared context and long hours are the symptoms of attention as the constraint.

The fixes at this stage look different. Fewer, better-scoped agent tasks. Agents that batch questions instead of interrupting. Tools that summarize what an agent did, so a person can review in minutes. Clear priorities, so engineers do not spend attention deciding what matters. The teams that solve this next will look less like they have more agents, and more like they ask their people for fewer decisions.

AI is necessary but not sufficient

If agents were enough on their own, every team would see Neil’s 11x. The data says they do not.

Atlassian’s State of Developer Experience 2026 surveyed more than 1,500 practitioners. Only 17% of developers and 23% of leaders said their organization has a clear, implemented AI strategy. Only 28% of developers feel less friction than before AI. Half, 49%, spend four or more hours a week reviewing AI output.

Faros’s April report is blunter: “Strong engineering foundations do not protect you.” Epics per developer rose 66%, but downstream quality got worse. Bugs per developer rose 54%.

Cortex’s 2026 benchmark found only 45% of organizations have AI policies. Its summary is the clearest single line on the subject: “AI amplifies both the best and worst aspects of your engineering practices.”

Charity Majors made the same argument in July. “Nondeterministic systems will require more engineering discipline, not less.”

Anthropic’s December study shows what the upside looks like when the organization adapts. 27% of Claude-assisted work at Anthropic was work that would not have been done otherwise. Running many Claude instances “allows you to test a bunch of different ideas.” That is experimentation built into the way people work.

So the evidence supports a simple rule. Agents are necessary for the large gains. They are not sufficient. The gains come from the changes around them: review moved to machines with guardrails, written context instead of tribal knowledge, systems rebuilt for agent pace, and limits on human attention.

Culture matters too. Teams that let people experiment with how they work find the next constraint sooner. Leaving agents running overnight, splitting work into workflows, and writing AGENTS.md files are all experiments that look odd at first. The organizations that allow them find out which ones work.

Speed is not product-market fit

Faster shipping helps a company only if it ships the right thing.

The speed is real. Stripe’s annual letter, published in February 2026, said its 2025 startup cohort is growing about 50% faster than the 2024 cohort. The number of companies reaching $10 million in annual revenue within three months of launch doubled. At Stripe Sessions in April, Stripe said top AI companies grew 120% in 2025 and 175% so far in 2026. Sam Altman said in April that the friction of starting a startup “has come down a lot.”

The retention is weaker. ChartMogul studied 3,500 companies for a December 2025 report. AI-native companies had a median gross revenue retention of 40% and net revenue retention of 48%. B2B SaaS had net revenue retention of 82%. ChartMogul’s summary: “The downside of being easy to buy is being easy to cancel.”

Marc Andreessen’s definition of product-market fit still frames the question: “being in a good market with a product that can satisfy that market.” Shipping more PRs does not answer it.

PostHog shows the order of events. It was founded in January 2020. By its own account, it reached product-market fit for its Scale product in September 2021, about 19 months after launch. Revenue grew six times in 2022. It passed 100,000 customers in October 2024 and turned cash-flow positive that December. Speed helped. Fit came first.

The edge for AI-native startups may also be temporary. Seema Amble of a16z wrote in September 2026 that “The incumbents aren’t sitting idly.” She argues that preserved context “is not a durable advantage.”

Established companies will adopt the same tools, more carefully, because they have more to lose. The startups that keep their lead will be the ones that turned speed into customers who stay.

A checklist for teams going AI-native

Here is what this edition adds up to, as checks a team can run this quarter.

Measure the change honestly.

• Track volume, PR size and rework together. Use follow-up fixes within seven days, not just reverts.

• Report per-developer change, not just totals.

• Know your cost per merged PR and per prevented incident.

Move review to machines, with guardrails.

• Route review depth by risk: a cheap reviewer on every PR, a deeper one on demand, humans on the riskiest tier.

• Give the reviewer repository search and a rules file that names shared modules.

• Turn every incident into a rule, and replay it against the PR that caused it before trusting it.

Encode the old human knowledge.

• Store money as integers in minor units, and test a zero-decimal and a three-decimal currency.

• Require concurrent index builds on large tables, in the reviewer’s rules and in a migration linter.

• Write AGENTS.md files for each area, so context outlives its owner.

Protect the scarce input: attention.

• Cap agent sessions per person, and finish before starting.

• Turn off agent notifications and check in on your own schedule.

• Set a hard end to the day. Do not make risky changes when tired.

Watch for the next constraint.

• List every system that assumes a human on the other end: CI, source control, rate limits, deploy windows.

• Redesign when the second patch lasts half as long as the first.

• Give one person ownership of design, so speed does not ship the default look.

The two eras are not separated by a model release. They are separated by which human steps a team removed, and how well it replaced them.

Reply to this email with two numbers from your team: merged pull requests per engineer per month, and median time from opening a pull request to merging it. We will publish what readers send, without names, in a future edition.

Sources

• Neil Kakkar, “Two eras of shipping” ledger, charts and table (neilkakkar.com)

• Neil Kakkar, “About” and CV (neilkakkar.com)

• GitHub, “Octoverse 2026”, 28 Oct 2026 (github.blog)

• Stack Overflow, “2026 Developer Survey results”, 6 Oct 2026 (stackoverflow.blog)

• Faros AI, “AI acceleration whiplash”, 12 Apr 2026 (faros.ai)

• Cortex, “2026 Engineering Benchmark” (cortex.io)

• Neil Kakkar, “Review ratio”, “Under the multiplier” and “What I built” charts (neilkakkar.com)

• Cloudflare, “AI code review”, 20 Apr 2026 (blog.cloudflare.com)

• Intercom, “AI is approving our pull requests”, 21 Apr 2026 (intercom.com)

• Anthropic, “Code Review for Claude Code”, 9 Mar 2026 (claude.com)

• Fortune, interview with Boris Cherny, 11 Jun 2026 (fortune.com)

• Google, “Sundar Pichai at Cloud Next 2026”, 22 Apr 2026 (blog.google)

• Claude Code docs, “GitHub Actions”, “Ultrareview” and “Customize code review” (code.claude.com)

• SIX, “ISO 4217 List One”, 17 Sep 2026 (six-group.com)

• Stripe docs, “Supported currencies” (docs.stripe.com)

• PostgreSQL docs, “CREATE INDEX” (postgresql.org)

• strong_migrations (github.com/ankane)

• GitLab docs, “Adding database indexes” (docs.gitlab.com)

• Squawk docs, “require-concurrent-index-creation” (squawkhq.com)

• Every, “Compound engineering”, 11 Dec 2025 (every.to)

• DORA, “DORA metrics guide”, updated 5 Jan 2026 (dora.dev)

• Google, “Site Reliability Workbook: Incident response” (sre.google)

• Amazon, 2015 letter to shareholders (sec.gov)

• PostHog, “Persons on events”, 2 Nov 2022 (posthog.com)

• Neil Kakkar, “AI made me braver”, 21 Jun 2026 (neilkakkar.com)

• Seedcamp, “Tano announces their angel round”, 24 Nov 2025 (seedcamp.com)

• Anthropic, “Building effective agents” (anthropic.com)

• Anthropic, “How AI is transforming work at Anthropic”, 2 Dec 2025 (anthropic.com)

• OpenAI, “Harness engineering”, 11 Feb 2026 (openai.com); InfoQ summary, 21 Feb 2026 (infoq.com)

• PostHog handbook, “Small teams”, “Product engineers” and “Helping engineers do product” (posthog.com)

• Neil Kakkar, “Agentic debt”, 24 Feb 2026, and “How I’m productive with Claude Code”, 16 Mar 2026 (neilkakkar.com)

• Anthropic, “Employment at Anthropic”, 16 Oct 2026 (anthropic.com)

• Simon Willison, “AI intensifies work”, 9 Feb 2026, and Lenny’s Podcast notes, 2 Apr 2026 (simonwillison.net)

• Steve Yegge, “The AI Vampire” (yegge-ai.pages.dev); Hanselminutes episode 1035, 5 Feb 2026 (hanselminutes.com)

• Harness, “State of DevOps Modernization 2026” (harness.io)

• Peter Denning, “Thrashing: its causes and prevention”, 1968 (cs.uwaterloo.ca)

• Neil Kakkar, “Work areas” and “Shape of a day” charts (neilkakkar.com)

• Addy Osmani, “Cognitive parallel agents”, 7 Apr 2026 (addyosmani.com)

• Harvard Business Review, “When using AI leads to brain fry”, 5 Mar 2026 (hbr.org); Fortune coverage, 10 Mar 2026 (fortune.com)

• PostHog handbook, “How we review” (posthog.com)

• Anthropic, “AI assistance and coding skills”, 29 Jan 2026 (anthropic.com)

• Charity Majors, “AI demands more engineering discipline, not less”, 27 Jul 2026 (oreilly.com)

• Avelino et al., “A novel approach for estimating truck factors”, 2016 (arxiv.org)

• AGENTS.md (agents.md); OpenAI Codex docs, “AGENTS.md” (learn.chatgpt.com)

• Mitchell Hashimoto, “My AI adoption journey”, 5 Feb 2026 (mitchellh.com)

• Anthropic, “Improving frontend design through skills”, 12 Nov 2025 (claude.com); “frontend-design” skill (github.com/anthropics)

• PostHog, “The first five” (posthog.com)

• Anthropic, “Claude pricing”, “Using Claude Code with your Pro or Max plan” and “API pricing” (claude.com)

• Tomasz Tunguz, “Tokenmaxxing”, 1 Apr 2026 (tomtunguz.com); IBM, “What is tokenmaxxing”, 1 Sep 2026 (ibm.com)

• PostHog, “Replay Vision”, 22 Sep 2026 (posthog.com)

• Sentry, “Automated debugging workflow”, 6 Aug 2026 (blog.sentry.io)

• Anthropic, “Agentic coding is straining CI”, 14 Sep 2026 (claude.com)

• Anthropic, “Redesigning source control”, 13 Oct 2026 (anthropic.com)

• TOC Institute, “The five focusing steps” (tocinstitute.org)

• Atlassian, “State of Developer Experience 2026” (atlassian.com)

• Stripe, “2025 annual letter” and “Stripe Sessions 2026: Inside the AI economy” (stripe.com)

• OpenAI Forum, Sam Altman event replay, 6 Apr 2026 (forum.openai.com)

• ChartMogul, “The AI churn wave”, 10 Dec 2025 (chartmogul.com)

• Marc Andreessen, “The only thing that matters” (pmarchive.com)

• PostHog handbook, “Our story” (posthog.com)

• a16z, “The Incumbents Are Coming”, 3 Sep 2026 (a16z.com)