Imagine asking an AI system to fix a bug and discovering that it has first appointed a product manager, an architect, a backend engineer, a tester, and someone whose main contribution is summarizing what the other four said.

The bug is still there. But it now has stakeholders.

This is the version of the agent-team pitch I find least convincing: take a capable model, divide its work among several copies with impressive job titles, and assume the resulting organization must be smarter than the original.

Maybe. But a reporting line is not an optimization.

My starting hypothesis is that some elaborate agent teams preserve workarounds for limitations that newer models handle better: fragile long conversations, muddled instructions, weak planning, and difficulty carrying a task across stages. That does not make every team obsolete. It makes the old justification something to retest, not inherit.

For work that needs one coherent view of a problem, I would start with one capable agent, a well-maintained library of specialist skills, and explicit checks. I would add separate agents only when separation produces a measurable benefit.

One owner. Many skills. Selective delegation.

That is a design preference supported by the research below, not a universal benchmark result. The evidence does not establish that one modern agent always beats a team. It does give us good reasons to stop treating the team as the premium package.

A specialist is not a job title

We need to separate three things that the sales diagram likes to put in the same box.

A persona tells the model how to approach or present its work: act as an architect, be a skeptical reviewer, prioritize accessibility.

A skill supplies reusable knowledge about doing the work: the procedure, relevant examples, scripts, constraints, and checks.

A separate agent runs its own decision loop, with its own working context and whatever tools the system gives it. LangChain’s documentation explicitly distinguishes on-demand skills, where one agent remains in control, from subagents and handoffs. These are different design choices, not different spellings of specialization. [1]

My objection is not to the word “architect.” It is to treating that word as evidence that another execution boundary is necessary.

Suppose the same underlying model is given the title “senior database engineer.” If we have supplied no additional database knowledge, tools, procedures, or evidence, what precisely have we specialized?

We have certainly specialized the introduction.

Now consider a database-migration skill that explains the project’s migration conventions, identifies the schema source of truth, includes validation commands, and specifies what to check before proposing a destructive change. That supplies something the agent can actually use.

Anthropic describes Agent Skills as packages of instructions, scripts, and resources that general-purpose agents discover and load as needed. The design separates having access to expertise from loading all that expertise into every prompt. [2]

That is the abstraction I would usually invest in first. Put the hard-won procedure somewhere reusable. Test it. Improve it when it fails. Let either the main agent or a delegated worker use it.

There is no need to make expertise permanently belong to a fictional employee.

This also changes what a meaningful specialist would look like. A separately trained model, a domain-specific solver, an unusual data source, or a genuinely different tool environment may justify a separate component. A different name above the same capabilities is a much weaker case.

Specialization should describe what the system can do differently, not what it calls itself before doing the same thing.

Some of the scaffolding really can go away

The idea that agent architectures can outlive the limitations they were built around is not just a convenient insult.

In its March 2026 report on long-running application development, Anthropic described removing forced context resets after moving from Sonnet 4.5 to Opus 4.5. With Opus 4.6, it also removed the sprint structure previously used to organize the build. It retained the planner and evaluator where they still helped; on tasks within the stronger generator’s capabilities, evaluation could become unnecessary overhead. These were engineering experiments, not a universal comparison across all coding work. [3]

That is the interesting lesson: an architecture can contain both temporary scaffolding and useful structure. Improving the model does not necessarily remove both.

I would therefore treat every extra stage as a claim: without this stage, some identifiable failure becomes more likely.

Perhaps the agent forgets a requirement. Perhaps it starts coding before resolving an ambiguous interface. Perhaps it produces a plausible-looking screen whose main interaction does not work.

Those are testable problems. “We need a planning department” is a staffing decision wearing a lab coat.

If a planning skill and a required implementation plan solve the first problem, keep them. If a fresh-context planner reliably improves the scope, keep that instead. The point is to preserve the useful behavior, not the original cast.

But there is a trap in taking the argument too far. Better models can also become better at delegation, verification, and integrating other agents’ findings. Improvement does not logically guarantee that single-agent systems win more often forever.

My inference is narrower: the right boundary moves as capabilities change. A workflow justified by last year’s failure trace should face this year’s test suite.

The expensive mistake is not building scaffolding. It is turning scaffolding into a listed building.

A skill library is not a prompt landfill

“One agent with lots of skills” needs an important qualification.

I mean a large pool of available capabilities, not a single prompt containing every procedure the organization has ever written, including the printer troubleshooting guide and a deeply emotional document about branch naming.

The active context should contain what this task needs.

The June 2026 revision of SkillsBench evaluated 87 tasks across 18 model–harness configurations: combinations of a model and the software that runs it. Curated skills increased the average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points. Tasks with focused bundles of one to three skills showed larger gains than those with bigger bundles. That does not establish a universal three-skill limit: the groups contain different tasks. Nor is SkillsBench a direct comparison of one skilled agent against a team. [4]

What it supports is more modest and more useful: well-chosen procedural help can matter, and more material is not automatically better.

I would organize a skill library around a small discovery layer: what each skill is for, when to use it, and what output it produces. Load the relevant procedure when needed. Keep bulky references and executable utilities separate. That follows the progressive-disclosure model in Anthropic’s skills design. [2]

Then keep task state separate from both the skills and the conversation: current requirements, decisions, unresolved questions, changed artifacts, and verification results. Anthropic’s context-engineering guidance treats selective retrieval, compaction, and persistent notes as complementary ways to manage long work. [5]

In this design, “one owner” means continuity of responsibility for the task. It does not mean one immortal chat transcript, one model call, or a heroic refusal to clear irrelevant output.

There is also a real counterargument here. LangChain’s worked examples show that isolated subagents can process fewer total tokens on some multi-domain tasks than a single agent that accumulates all the domain instructions. Those examples illustrate the tradeoff; they are not universal production measurements. [1]

So I would measure the active context and the bill, rather than assume fewer agents always means fewer tokens.

A library should help you find the right book. It should not require you to swallow the building.

The task matters more than the team size

The strongest reason to distrust blanket claims is that the experimental results point in both directions.

The April 2026 revision of Towards a Science of Scaling Agent Systems compared 260 configurations across six benchmarks and five architectures. Relative to single-agent baselines, reported performance changes ranged from a gain of 80.8% on decomposable financial reasoning to a decline of 70.0% on sequential planning. The authors’ most robust finding was diminishing benefit from coordination as single-agent baselines became stronger. [6]

That is not “multi-agent works” or “multi-agent fails.” It is a warning about matching coordination to the work.

The study’s aggregate SWE-bench Verified results also favored the single-agent baseline, but that evaluation used only 20 instances per configuration, with wide uncertainty. It is not a verdict on every repository, modern model, or purpose-built coding team. [6]

Here is how I would translate the evidence into a working decision.

Can the subtasks make useful progress without repeatedly renegotiating the same assumptions?

If three investigations can each return an independent piece of evidence, parallel workers have a sensible job. If every next step depends on the evolving state of the previous one, splitting responsibility may mostly create more opportunities to misunderstand that state.

Consider a hypothetical chain: one agent interprets a requirement, another turns the interpretation into an interface, a third implements it, and a fourth tests the implementation against the interface. If the original interpretation was wrong, the chain can produce beautifully consistent evidence that it built the wrong thing.

The problem is not the number four. It is the absence of a check against the original requirement.

Equally, keeping everything with one agent does not make the first interpretation correct. It merely removes some translation boundaries. Verification still has to happen.

I would rather draw the dependencies first and decide the agent boundaries afterward.

The org chart is the output of that exercise, not the input.

Delegation should buy something specific

There are three reasons I would readily add another agent: useful parallel work, a separate evidence-producing check, or a boundary that needs to be real.

Parallel work buys time or coverage.

Anthropic reported that its June 2025 research system, using an Opus 4 lead and Sonnet 4 workers, outperformed a single Opus 4 agent by 90.2% on its internal research evaluation. It particularly emphasized breadth-first investigations. The report also described much higher token use: about 15 times ordinary chat for multi-agent systems, versus about four times for agents. Those are comparisons with chat, not a claim that teams used 15 times the tokens of a single agent. [7]

That result is a serious counterexample to “teams are just obsolete scaffolding.” It is not an equal-budget proof that teams win everywhere.

For a broad investigation, I would happily send separate workers down distinct evidence trails and ask each for sources, findings, and unresolved contradictions. The lead can then concentrate on synthesis.

But I would not create a new agent just to run an independent command. A main agent can initiate parallel tool work too; LangChain explicitly distinguishes that from multi-agent coordination. [1]

Run the tests concurrently when appropriate. They do not need individual career paths.

A separate check buys a different opportunity to catch an error.

The valuable reviewer is not merely the one instructed to be grumpy. It is the one that receives the original acceptance criteria, inspects the actual artifact, and brings back something verifiable.

In the March harness report, Anthropic’s evaluator exercised applications through Playwright and returned concrete defects. The author also reported needing to tune the evaluator: a separate agent could still test superficially or excuse problems. Separation alone was not enough. [3]

For my own workflow design, I would give a reviewer room to form an initial judgment before seeing the builder’s defense of the work. I would ask for a failing test, a reproducible interaction, or a specific mismatch with the requirement.

That is an attempt to create evidential independence. A fresh context is not proof of statistically independent errors, particularly when both agents use the same model. Agreement is not a substitute for checking the artifact.

A real boundary buys control.

Suppose an exploratory worker can read untrusted material, while another component can approve a production action. I would keep the relevant permissions separate and enforce that separation in the runtime.

A skill saying “please be careful” is not an access-control mechanism. Conversely, another agent with the same unrestricted credentials has not created a meaningful boundary either.

The same reasoning applies to a worker with genuinely different tools or a separately evaluated specialist model. Keep it separate when the capability or constraint warrants separation—not simply because a diagram has an empty box.

Notice what is absent from these reasons: sounding like a company.

What this looks like on a real ticket

Take an illustrative TypeScript booking service. Two concurrent requests can both claim the last available place.

This is a hypothetical workflow, not a report of a production experiment.

A fixed-team approach could route the ticket through a planner, architect, database agent, API agent, tester, and reviewer. It could work. Before choosing it, I would want to know which of those handoffs solves a problem that one owner with the same knowledge cannot solve.

Here is the simpler candidate I would test.

First, establish the invariant. The owner reads the requirement and existing behavior, then records what must remain true: no oversubscription, no duplicate reservation from a retry, and a defined response for the request that loses the race. Existing authorization and API behavior remain constraints, not optional accessories.

That is the task contract. Every later decision and check must refer back to it.

Next, load the relevant capabilities. The owner uses the repository’s database-concurrency procedure, API conventions, and integration-test guidance. It traces the actual request path, examines how state changes are committed, and creates a reproducer before deciding on the fix.

The agent is not expected to guess the project’s conventions from its job title. Nor should it load the entire deployment manual unless the change touches deployment.

Delegate a bounded investigation only when useful. A worker might inspect other reservation entry points while the owner studies the main path. Give it a question, a code snapshot, read-only scope, and a required output: file locations, relevant behavior, and evidence of any path that can bypass the proposed protection.

Do not ask it to “be the architect.” Ask it something it can finish.

Keep integration with the owner. The owner decides how the findings affect the patch and updates the task record. If multiple workers must change code, specify their write boundaries and integration order; do not leave them negotiating ownership through overlapping edits.

Verify the resulting artifact. Run the reproducer, the relevant tests, and the agreed regression checks. A separate reviewer can then inspect the patch against the original invariant and exercise a less obvious scenario: retries, cancellation, another request path, or an unexpected failure between operations.

The useful output is the scenario and evidence, not a ceremonial approval.

Finally, the owner reports what changed, which checks ran, which passed, what remains uncertain, and whether the work was actually committed or merely described with tremendous confidence.

This design leaves a human or organization responsible for the release decision. Calling an agent the “owner” does not outsource accountability to software.

And yes: once the owner launches a worker or reviewer, this is technically a multi-agent system.

That is not a loophole. It is the point. I am arguing against a mandatory cast of permanent roles, not against invoking another model when there is a good reason.

The stable unit is the task and its evidence. The staffing is temporary.

Make the architectures compete, not the demos

To choose between these designs, I would run three candidates on the same representative tasks: a strong single agent with curated skills, a fixed specialist team, and an owner that can delegate selectively.

Give them equivalent access to the requirements, knowledge, tools, and checks. Let the team use the skills too. Otherwise the comparison is partly measuring whether one side was allowed to read the manual.

I would include sequential bug fixes, decomposable investigations, multi-file changes, and work that demands independent verification. Freeze the starting artifacts, record the model and skill versions, and repeat runs rather than choosing the most flattering transcript.

Two comparisons matter. Under an equal total spending cap, which system produces more accepted work? Under the same deadline, how much quality does each deliver, and what does it cost?

Those are different questions. More parallel work can be worth paying for when time matters. It should not quietly be presented as a free architectural improvement.

My scorecard would prioritize correctly completed tasks, regressions, missed requirements, time to an acceptable result, total cost, and human intervention. I would count the clarification, integration, and repair work too. A handoff that saves agent time by creating human cleanup has not disappeared from the bill.

Then remove components one at a time. Does removing the planner reduce quality? Does moving review to the end miss defects? Does replacing a permanent specialist with an on-demand skill change outcomes? Does adding one bounded worker actually shorten the critical path?

Anthropic’s broader guidance has long recommended starting with the simplest workable design and increasing complexity only when justified. It also distinguishes predefined workflows from agents that choose their own next actions. Some steps need a reliable function or a fixed check, not another autonomous participant. [8]

I would apply that discipline to skills as well. A long, stale procedure can deserve deletion just as much as a redundant agent.

Preference is not performance. Neither is a lively coordination transcript.

Keep the expertise. Make the headcount earn its place.

My default would be a capable owner with a small active context, reusable skills, durable task state, and checks that produce inspectable evidence.

I would add workers for genuinely separable work, reviewers when they improve verification, and separate components when capabilities or permissions demand it. I would retest those choices when the model, task mix, or tool environment changes.

That is not the claim that one agent can do everything. It is the refusal to assume that every useful capability needs its own employee.

If a second agent improves the result, keep it. If a sixth agent mainly explains what the other five are doing, check whether you have automated the work or merely reproduced the meeting.

Before adding another agent, name what it will contribute that the current system cannot supply as a skill, a tool, or a check.

If the answer is a job title, the org chart is doing more work than the architecture.

* * *

Notes and References

Evidence checked October 4, 2026. Benchmark percentages describe the cited evaluations, not guaranteed gains on other workloads. The recommended architecture and booking workflow are the author’s synthesis and proposal; no head-to-head production experiment is claimed here.

1. LangChain. “Multi-agent.” Documentation, accessed October 4, 2026. Distinguishes skills, subagents, handoffs, and their context/cost tradeoffs.

2. Anthropic. “Equipping agents for the real world with Agent Skills.” October 16, 2025; open-standard update December 18, 2025.

3. Prithvi Rajasekaran, Anthropic. “Harness design for long-running application development.” March 24, 2026. Engineering case studies; model and harness changes are not a controlled, equal-budget architecture trial.

4. Xiangyi Li et al. “SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks.” arXiv:2602.12670, version 4, June 14, 2026. Terminal-based benchmark; skill-count groups do not isolate the causal effect of adding another skill.

5. Anthropic. “Effective context engineering for AI agents.” September 29, 2025. Guidance on relevant context, compaction, and persistent notes.

6. Yubin Kim et al. “Towards a Science of Scaling Agent Systems.” arXiv:2512.08296, version 3, April 8, 2026. Benchmark-dependent findings; small coding subsets limit individual comparisons.

7. Anthropic. “How we built our multi-agent research system.” June 13, 2025. Internal evaluation using Claude 4 models; reported token multipliers use ordinary chat as the baseline.

8. Anthropic. “Building effective agents.” December 19, 2024. Cited for design principles, not as an inventory of current tools.

Prompt engineering contains a small corporate ritual that has survived an impressive number of model generations: before asking the model to do anything, promote it.

You are a world-class lawyer. You are a senior principal software architect. You are an elite McKinsey strategist with twenty years of experience in distributed systems, paediatric cardiology and, apparently, whatever else is in the next paragraph.

The model has no medical licence, no bar membership, no production pager, and no scar tissue from the migration that went wrong at 2:17 a.m. But congratulations. It has been promoted.

I think of this as cosplay prompting: giving the model an impressive identity and hoping the costume improves the work.

There is now a counter-claim floating around prompt-engineering discussions: this used to work, but modern models are too capable for it. Just ask directly. The cape is obsolete.

That correction is useful, but it is still too neat. The research does not show a clean moment when persona prompting expired. It shows something more annoying: the effect depends on the task, the model, the exact wording, the baseline, and what you measure. Expert labels are not a dependable accuracy upgrade. Roles can still change behaviour, and sometimes performance. Those are different claims.

The practical rule is simpler than the debate: give the model a job, not a résumé.

The prompt did not install a career

The phrase “role prompting” bundles together several things that should never have been treated as one technique.

• Credential cosplay — “You are a world-class tax lawyer.”

• Functional responsibility — “Review this contract for termination, liability and renewal risk.”

• Audience framing — “Explain this to a CFO who understands finance but not distributed systems.”

• Behavioural protocol — “Separate verified facts from assumptions; cite evidence; flag uncertainty.”

• Simulation — “Respond as a buyer evaluating this onboarding flow.”

Only the first one claims, implicitly, that an identity label should make the model more knowledgeable or accurate. The others specify what work to do, what perspective to use, or how to communicate it.

That distinction matters because a model cannot acquire new domain knowledge from a flattering sentence about itself. A role prompt can change which patterns it samples, which concepts it foregrounds, how much jargon it uses, what risks it notices, or how it structures an answer. That can be useful. It is not the same as installing ten years of professional judgment with one line of text.

Current vendor guidance is actually more nuanced than the folklore. OpenAI still discusses role and identity guidance, but also tells developers to treat prompts as code, pin model versions and run evaluations because behaviour can change between model snapshots. Anthropic still recommends giving Claude a role, alongside clear instructions, context and examples. Google still lists persona or role as one optional component of a prompt, next to the actual task, constraints, context and output format.[7][8][9]

In other words, the vendors have not declared role prompting dead. They have quietly put it back where it belongs: one control among several. The industry then did what it often does with a nuanced control surface and turned one knob into a religion.

Why the costume got a reputation

It is worth remembering that role-play prompting did produce striking results in earlier research. This was not invented entirely by people selling prompt templates on social media.

A 2024 NAACL paper by Aobo Kong and colleagues tested a strategically designed role-play prompting method across twelve reasoning benchmarks. On ChatGPT, accuracy on AQuA rose from 53.5% to 63.8%. On the Last Letter task it jumped from 23.8% to 84.2%. The authors also found their role-play setup could act as a stronger trigger for chain-of-thought-style reasoning than a standard zero-shot baseline.[1]

Those are not rounding errors. If you saw results like that, you would reasonably conclude that role framing could matter.

But notice what the result does not establish. It does not establish that attaching prestige to any arbitrary task makes any future model smarter. It shows that a particular role-play intervention, on particular models and benchmarks, changed the model’s reasoning behaviour enough to improve results.

This is where prompt advice tends to go through its familiar laundering process. A paper finds a conditional effect. A blog turns it into a technique. A slide turns the technique into “best practice.” Six months later somebody is beginning a CSV-cleaning prompt with “You are the world’s foremost data scientist.”

Which would be harmless, except teams then confuse the costume with the mechanism.

Then the results stopped behaving

A different 2024 study gives the opposite impression. Mingqian Zheng and colleagues tested 162 personas across four model families and 2,410 factual questions. Adding personas did not improve performance overall compared with no persona. Gender, role type and domain could change results, but the effect was inconsistent. The researchers could identify personas that worked better for individual questions after the fact; reliably choosing the best persona in advance was much harder, with automatic strategies often doing no better than random selection.[2]

That is a fairly devastating result for “always start with an expert role.” A prompt trick that only works once you already know which costume wins is not much of a production strategy.

Then the 2025 EMNLP paper Principled Personas made the picture messier again. It evaluated nine state-of-the-art open models over 27 tasks. Expert personas usually caused positive or statistically non-significant changes, and Llama 3.1 70B with dynamic focused-expert personas showed strict improvement on 37% of tasks. But the same study found models surprisingly sensitive to irrelevant persona details, with drops of almost 30 percentage points in some settings.[3]

Read that carefully. “Personas do nothing” is wrong. “Personas reliably help” is also wrong. Sometimes an expert frame helps. Sometimes adding irrelevant decoration damages the result. The model is steerable, but not in the tidy way a prompt cheat sheet would like.

A 2025 Wharton Generative AI Labs report pushed specifically on the prestige claim. Six models were tested on GPQA Diamond and an MMLU-Pro subset using in-domain experts, mismatched experts and deliberately low-knowledge personas. Expert personas did not consistently improve factual accuracy; Gemini 2.0 Flash on MMLU-Pro was the notable exception. Low-knowledge personas were generally harmful, and mismatched roles could also degrade behaviour.[4]

The serious point underneath the comedy is that persona text can move the model. The problem is that “it moved” is not the same as “it got better.”

“Works” is doing far too much work

A lot of the argument disappears once you ask what “works” means.

If the metric is factual accuracy, “You are an expert” has weak support as a general technique. If the metric is tone, terminology, depth, perspective, empathy, critique style or audience fit, a role can be doing exactly what you want even when accuracy stays flat.

A 2026 preprint makes this distinction unusually visible. Across 1,140 open-ended questions, 38 expert roles and six domains, the authors found only small aggregate differences between persona conditions. But underneath the average, role prompting tended to increase perceived expertise depth while reducing clarity. The persona was not simply adding quality. It was trading one quality dimension for another.[6]

That sounds obvious once stated. Ask a model to sound like a specialist and it may produce more specialist language. Congratulations: the cardiologist costume came with cardiologist prose. Whether the reader needed cardiologist prose is a separate product decision.

Persona prompting is even more clearly a different problem when the persona is meant to simulate people rather than expertise. An EACL 2026 study found demographic persona prompting could improve classification on a highly subjective hate-speech task while degrading rationale quality; the simulated personas also failed to align reliably with the corresponding real demographic groups.[5]

So “respond as a 55-year-old nurse from X” should not be treated as cheap user research. A model can produce a plausible character. Plausibility is not representation. This is the same trap in a different costume.

There is no proven expiration date

The tempting modern story is that cosplay prompting worked on older, weaker models and stopped working on newer reasoning models.

I would not make that claim from the evidence we have.

The studies above are not a clean longitudinal experiment where the same prompt intervention, task set and evaluation are run across model generations until the effect vanishes. They use different models, different persona constructions, different benchmarks and different metrics. The sensible conclusion is not “role prompting died in 2025.” It is that there was never a universal role-prompting law to begin with.

Modern models do change the economics. OpenAI’s current guidance for reasoning models explicitly recommends straightforward prompts and warns that older incantations such as asking the model to “think step by step” may be unnecessary or counterproductive.[10] That should make us suspicious of any ritual text we keep copying forward merely because it once helped a different model.

But “less necessary” and “invalid” are not the same thing. If a current model already understands the task, a vague expert title may add little. If the role compresses useful information about responsibility, audience or decision criteria, it can still steer the answer in a useful direction.

The real change is that prompt engineering is becoming less like spell casting and more like software configuration. Specify the behaviour you need. Test it against a baseline. Keep it only if the measured outcome improves.

Preference is not performance. Tradition is definitely not performance.

A code review without the cape

Suppose a team wants an AI coding agent to review a legacy payment-handler change before merge.

The cosplay version looks familiar: “You are a world-class principal engineer and payments expert with twenty years of experience in distributed systems and security. Perform an expert review.”

It may produce a good review. It may also produce a very confident essay about dependency injection. The prompt has told the model how impressive it is, but very little about what would make this review useful.

Now remove the biography and give it a job.

“You are the merge-gate reviewer for this payment change. Decide whether it is safe to merge. Check duplicate-charge risk, idempotency, transaction boundaries, retries, concurrency, authorization, secret handling, rollback behaviour and missing tests. For every issue, cite the code evidence and explain the concrete failure mode. Separate confirmed defects from hypotheses. Do not propose refactors unless they address a demonstrated risk. Finish with Block, Warn or Pass and the reasons.”

The second prompt still contains a role. But the role is operational. It defines responsibility, scope, evidence, uncertainty handling and the decision to produce.

Imagine the model spots that a retry path can call the charge operation twice unless an idempotency key is stable. Useful. It then claims the ORM automatically rolls back a nested transaction in a way that sounds plausible but is wrong for the version you run. Also normal.

The verification step is not “ask the model if it is sure.” You check the transaction implementation, the ORM documentation and the tests. If the risk matters, you reproduce it. A title is not a unit test.

This is the operating lesson: use prompting to focus judgment, not to manufacture authority. The model can help widen the search, structure the review and surface hypotheses. Evidence still decides whether the hypothesis survives.

In practice, this is also easier to evaluate. You can run the same patch through the cosplay prompt and the operational prompt, then compare defect recall, false positives, citation quality and review time. Now you are doing engineering rather than prompt astrology.

Where roles still earn their costume

I would still use roles. I would just stop expecting the role name itself to carry the system.

Roles are useful when they compress a real behavioural contract. “Security reviewer” can mean prioritize exploitability and trust boundaries. “Editor for a technical audience” can mean preserve technical detail while cutting repetition. “Product manager preparing a launch review” can mean surface dependencies, unresolved decisions and measurement gaps. The useful content is the implied work pattern; when it matters, make that pattern explicit.

Roles are also useful when the goal is genuinely perspectival. Asking for a skeptical reviewer, an impatient first-time user, or a regulator’s likely line of questioning can help generate attack angles you might not have considered. That is ideation, not evidence. The output becomes a queue of things to verify, not a synthetic substitute for the people being simulated.

And if your only goal is tone, cosplay is fine. Tell the model to write like a patient tutor, a terse incident commander, or a mildly annoyed staff engineer. Style is exactly the thing a style instruction is allowed to change.

What I would not do is use “You are an expert” as the control that is supposed to make a high-stakes answer correct. For correctness, give the model better context, access to authoritative sources, tools, examples, constraints and a verification path. Then evaluate the result on the task you actually care about.

If the expert label survives that evaluation, keep it. If removing it changes nothing, delete it. If it makes the model sound more authoritative while accuracy stays flat, delete it faster.

The annoying question

So, is cosplay prompting still valid?

Yes, as steering. No, as a general intelligence upgrade.

There is no solid evidence that modern models crossed some threshold after which roles stopped mattering. There is strong evidence that expert-persona effects are conditional, sometimes negligible, sometimes positive, sometimes harmful, and easy to confuse with changes in style or depth.

The useful question is not “Should I tell the model it is an expert?” It is “What behaviour am I trying to cause, and can I specify and measure that behaviour directly?”

If “You are a world-class engineer” is shorthand for a review process you have tested and it improves outcomes, fine. Keep the cape.

If it is there because somebody copied a prompt from 2023 and nobody wants to touch the sacred paragraph, that is not prompt engineering. That is a ritual with better formatting.

Give the model the responsibility, the evidence, the constraints and the test. Let the résumé stay fictional.

Notes and References

1. Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, Xiaohang Dong, Better Zero-Shot Reasoning with Role-Play Prompting, NAACL / Association for Computational Linguistics, 2024. Used for the twelve-benchmark role-play study and the reported AQuA and Last Letter accuracy changes.

2. Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, David Jurgens, When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models, Findings of EMNLP / Association for Computational Linguistics, 2024. Used for the 162-persona, four-model-family, 2,410-question evaluation and the difficulty of selecting a reliably beneficial persona.

3. Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth, Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance, EMNLP / Association for Computational Linguistics, 2025. Used for the nine-model, 27-task evaluation, the 37% strict-improvement result for focused dynamic experts on Llama 3.1 70B, and sensitivity to irrelevant persona attributes.

4. Savir Basil, Ina Shapiro, Dan Shapiro, Ethan Mollick, Lilach Mollick, Lennart Meincke, Prompting Science Report 4: Playing Pretend: Expert Personas Don’t Improve Factual Accuracy, Wharton Generative AI Labs, 2025. The report was revised in 2026; the underlying study is dated December 2025. Used for the six-model GPQA Diamond and MMLU-Pro experiments showing no consistent factual-accuracy gain from expert personas and generally harmful effects from low-knowledge personas.

5. Jing Yang, Moritz Hechtbauer, Elisabeth Khalilov, Evelyn Luise Brinkmann, Vera Schmitt, Nils Feldhus, Persona Prompting as a Lens on LLM Social Reasoning, EACL / Association for Computational Linguistics, 2026. Used for the finding that persona prompting could improve classification on a subjective hate-speech task while degrading rationale quality and failing to reproduce real demographic alignment.

6. Shuai Xiao, Su Liu, Weikai Zhou, Jialun Wu, Xinjie He, Zhiyuan Lin, Qiyang Xie, When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs, arXiv preprint, 2026. Preprint; not treated here as peer-reviewed evidence. Used for the 1,140-question analysis suggesting a trade-off between expertise depth and clarity rather than a broad aggregate quality gain.

7. OpenAI, Prompt engineering, OpenAI API documentation, 2026. Used for current guidance on role/identity instructions, model-specific prompting, snapshot variability, prompt versioning and evaluation.

8. Anthropic, Prompting best practices, Claude Platform documentation, 2026. Used for current guidance to use clear and direct instructions, examples and system-role framing where it helps focus behaviour and tone.

9. Google Cloud, Overview of prompting strategies, Vertex AI documentation, 2025–2026. Used for the framing of persona/role as one optional prompt component alongside task, constraints, context, examples and output format.

10. OpenAI, Reasoning best practices, OpenAI API documentation, 2026. Used for the current recommendation that reasoning models generally perform best with straightforward prompts and that explicit chain-of-thought requests such as “think step by step” may be unnecessary or counterproductive.

The first day with a new AI model can feel extraordinary.

You give it something the previous model struggled with. It understands the request, makes a useful connection, and produces an answer that needs surprisingly little correction. You try another task. That works too.

Finally, you think. This is the one.

Then we do what organisations have always done when something proves competent.

We give it more work.

The requests become more ambitious. The neat little test prompts disappear. We bring it into actual work, where the requirements are unfinished, the evidence conflicts, three people have different ideas about what “done” means, and getting something wrong has consequences.

Before long, you are correcting it, repeating yourself, and wondering why it keeps missing the point. Meanwhile, the usage allowance that looked perfectly reasonable when you were asking it to tidy paragraphs is now disappearing with the urgency of a departmental budget in the final week of the financial year.

The obvious explanation is that the model has been “dumbed down.”

Sometimes the product really has changed. In April 2025, for example, OpenAI acknowledged that an update had made GPT-4o excessively agreeable and rolled it back. Genuine regressions should not be dismissed as users imagining things. ¹

But a model does not have to deteriorate for our experience of it to deteriorate.

There is another explanation worth considering: success earns the model more trust, trust earns it a bigger job, and the bigger job exposes limitations we had not previously encountered.

We may be doing to AI what organisations have long done to their best employees.

The model passes the test. Then we change the test.

Consider the difference between asking an AI to improve a paragraph and asking it to produce a publishable article.

The first task has a relatively clear boundary. The words are already there. You can judge whether the revision is better.

The second requires much more: a defensible argument, accurate evidence, an understanding of the audience, appropriate emphasis, and enough judgement to know what should be left out.

Now extend the request again. Ask it to research the subject, choose the argument, write the article, adapt it into a campaign, and ensure everything fits your organisation’s positioning.

It may still look like a writing request.

It is no longer the same job.

This is the bit we are strangely good at ignoring. The interface has not changed. There is still a box. We still type words into the box. Therefore, apparently, everything involving words is one task.

The same distinction applies to software. Fixing a clearly identified function is different from diagnosing an intermittent failure across an application. Producing working code is different from changing a system without breaking assumptions elsewhere.

There is experimental evidence for this unevenness. In a study involving 758 Boston Consulting Group consultants, AI assistance helped participants complete suitable tasks about 25% faster. Yet on a task selected to fall outside the model’s capabilities, AI-assisted participants were 19 percentage points less likely to reach the correct answer than those working without it. The researchers described a “jagged technological frontier”: success on one task did not reliably establish competence on another. ²

That study does not prove that launch-day enthusiasm fades because users assign harder work. It establishes something narrower and important: changing the assignment can reverse the value of using the same AI.

A model can therefore remain just as capable while becoming less dependable for the work we now expect it to do.

On the first day, the question was:

“Can it help me with this?”

Later, the question becomes:

“Why can’t it take care of this without me?”

Those are different standards.

We just tend to promote the model between them without mentioning it.

The Peter Principle, at software speed

The Peter Principle describes the tendency to promote people on the strength of their current performance until they reach a role in which they are no longer effective. Research by Alan Benson, Danielle Li, and Kelly Shue, using data from 131 firms, found evidence of companies favouring strong sales performers for promotion even when other characteristics better predicted managerial performance. ³

The important point is not that successful people somehow become stupid.

It is that the new role may require different abilities. Being excellent at winning customers does not automatically establish that someone can coach a team, allocate resources, or resolve conflict.

The official version is that we recognised excellence and rewarded it.

The less flattering version is that someone was very good at Job A, so we stopped letting them do Job A.

We can make the same mistake with AI, only much faster.

It summarises documents well, so we promote it to analyst. It produces a convincing analysis, so we promote it to adviser. It gives useful advice, so we ask it to make decisions and carry out the work.

No interview. No probation period. No awkward conversation about whether it actually wants management responsibilities. One good Tuesday afternoon and congratulations: you now own the workflow.

Each promotion feels justified by the previous success. Yet each may involve an untested jump.

A model that helps you think is not necessarily ready to decide on your behalf. A model that can complete a task with close guidance is not necessarily ready to own an entire workflow.

Call this an AI version of the Peter Principle: we keep expanding the model’s responsibilities until we find the work it cannot reliably handle.

Then we judge it by that work, rather than by the capabilities that earned our trust in the first place.

Promotion and overload are different problems

There is a second organisational mistake hiding inside this pattern.

Sometimes a person is given the wrong job.

Sometimes they have the right skills but are given too much work.

Imagine a dependable employee who handles difficult assignments well. Whenever something urgent appears, it goes to them. Whenever another team falls behind, they help. Their reward for being reliable is an expanding collection of responsibilities.

This is one of management’s more durable operating systems: if someone reliably carries things, find more things.

Eventually, something slips.

Calling this incompetence would confuse ability with workload. Burnout is not proof that someone has reached the limit of their talent. The World Health Organization describes it as a consequence of chronic workplace stress that has not been successfully managed, involving exhaustion, detachment or cynicism, and reduced professional efficacy. ⁴

That distinction matters when drawing the AI comparison.

Models do not burn out in the human sense.

They do not become exhausted, cynical, or sit through a Sunday evening wondering how “one small additional responsibility” somehow became twelve.

But they do operate within limits—and those limits are not all the same.

One is capability: whether the model can reliably perform the task.

Another is context: how much relevant information it can use effectively during the work. A longer conversation can accumulate instructions, corrections, discarded approaches, and tool results. Anthropic’s guidance on context engineering explicitly warns that model performance can degrade as context grows; a larger context window does not guarantee equally effective use of everything inside it. ⁵

A third is the usage allowance: how much work the service permits before access is restricted or additional usage must be purchased. That is separate from the length of an individual conversation. ⁶

These are different diagnoses.

A difficult task does not become easy because your allowance resets. Buying more usage does not automatically improve judgement. Starting a cleaner conversation may reduce contextual clutter, but it does not supply a capability the model lacks.

This sounds obvious when written down. In practice, these problems are often experienced through the same highly technical diagnostic instrument:

“It was better yesterday.”

The analogy with people is therefore useful only up to a point. Human burnout requires attention to working conditions, support, and recovery. AI requires an appropriate task, usable context, and sufficient resources.

The shared mistake is treating demonstrated competence as evidence of unlimited capacity.

Why the usage allowance seems to disappear faster

This also offers a plausible explanation for the feeling that each new release gives us less usable time.

Think about what happens after a model earns your confidence.

You stop using it only for occasional questions. You give it longer documents, larger projects, and work you previously would not have delegated. You ask it to investigate, compare, revise, verify, and try again.

The model has proved that it can carry boxes.

Naturally, we respond by locating every box in the building.

You may type fewer messages while requesting substantially more processing.

Anthropic’s usage documentation states that consumption depends on factors including conversation length and complexity, the model, the features used, and the selected effort level. A message is therefore not a fixed unit of work. ⁶

With an agent, the gap between what you type and what the system does can be particularly large. A short instruction may initiate repeated file reads, tool calls, edits, and checks. Claude Code’s documentation explains that tool interactions generate further requests and that long conversation histories continue contributing to usage, even when caching reduces their cost. ⁷

Reasoning can add another layer. OpenAI’s API documentation notes that reasoning tokens count towards output usage even though they are not displayed as ordinary answer text. A short visible response need not represent a small amount of processing. ⁸

This creates a counterintuitive possibility: a more useful model can make the same allowance feel smaller because you have found more valuable—and more demanding—ways to spend it.

That is not evidence that every newer model consumes more resources, or that providers never change their limits. It means that reaching a cap sooner does not, by itself, establish that the allowance was reduced.

The useful comparison is not simply how many prompts you sent.

It is how much verified work you completed, including the effort spent correcting unsuccessful attempts.

A model that uses more resources but reliably completes the job may be better value. One that spends the allowance repeatedly failing may not be.

Preference is not performance. Neither is prompt count.

Why the honeymoon can get shorter

The first time you encounter a capable AI assistant, you may approach it cautiously.

You experiment with small tasks. You discover what it does well. You learn where it gets confused. You gradually find the boundary between “surprisingly useful” and “why did I let it touch this?”

A later model does not necessarily receive that gentle introduction.

It inherits your existing workflows, your accumulated expectations, and the backlog of work the previous model could not handle.

Perhaps the last model could draft a report but struggled to reconcile conflicting evidence. The next one demonstrates better analysis, so you immediately give it the whole research assignment.

Perhaps the previous coding agent needed constant supervision. The new one solves a difficult bug, so you immediately ask it to work independently across the project.

Yesterday’s breakthrough becomes today’s minimum requirement remarkably quickly.

This suggests a reason the honeymoon can shorten without the technology declining faster: we arrive at each release better prepared to reach its limits.

The capability improvement may be real. But instead of continuing to assign yesterday’s work and enjoying the improvement, we spend that improvement on greater ambition.

This is a hypothesis about the experience, not a measured rule governing every release.

It may not explain your experience with a particular model at all. Products can change. Limits can change. Models differ in capability, context behaviour, and usage allowances. Genuine regressions happen.

But the hypothesis explains how two impressions can both be sincere: the model really was impressive at first, and it really is struggling now.

What changed may be the distance between the task and the model’s dependable capabilities.

We did not necessarily discover that the new employee was secretly terrible.

We may simply have completed the promotion cycle unusually efficiently.

Before blaming the performer, inspect the assignment

The practical response is not to stop trusting AI or to keep capable people permanently in small roles.

It is to make expansion deliberate.

For AI, preserve a small set of representative tasks, with the original inputs and clear standards for success. When a model seems worse, repeat those tasks under comparable conditions, preferably in fresh conversations. Repeat enough times to avoid treating one unusually good or bad answer as the whole story.

This gives you a better basis for separating a genuine regression from a harder workload, a cluttered conversation, or a higher standard of success.

Then test new responsibilities as new responsibilities.

Strong drafting is evidence for drafting.

Reliable analysis is evidence for analysis.

Neither should silently become permission for unrestricted decision-making.

The same applies when the tool moves from answering questions to operating inside a workflow. If the job now includes reading files, choosing an approach, editing code, checking the result, recovering from errors, and deciding when the work is finished, you have changed more than the prompt.

Treat it accordingly.

For people, apply the same discipline without forgetting the human difference. Before deciding that someone has lost their edge, examine how their role has changed, what has been added, what support has disappeared, and whether the workload is sustainable.

A promotion should come with an assessment of what the new role requires. Additional responsibility should come with an explicit decision about what will be removed, deferred, or supported.

Otherwise, “more responsibility” has a tendency to mean “everything you already do, plus this”.

Success should justify greater trust.

It should not remove the need for boundaries.

The next time a brilliant new AI seems to have become disappointing, ask two questions rather than one: has the system changed, and has the job changed?

And when a once-dependable employee begins to struggle, ask those questions before reaching for a label.

Sometimes capability has declined. Sometimes the match was wrong. Sometimes the workload has become unreasonable.

But sometimes the performer is still capable of everything that first impressed us.

We have simply stopped asking for that—and started asking for everything else.

Notes and References

1. OpenAI, Sycophancy in GPT-4o: What happened and what we’re doing about it, OpenAI, 2025. Used for the acknowledged GPT-4o regression and rollback.

2. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality, Organization Science, 2026. Used for the experiment involving 758 consultants, faster completion within the frontier, and worse accuracy outside it.

3. Alan Benson, Danielle Li and Kelly Shue, Promotions and the Peter Principle, The Quarterly Journal of Economics, 2019. Used for the evidence from 131 firms on promotion and managerial performance.

4. World Health Organization, Burn-out an occupational phenomenon, WHO. Accessed 30 September 2026. Used for the distinction between human burnout and model limits.

5. Anthropic, Effective context engineering for AI agents, Anthropic. Accessed 30 September 2026. Used for the limits of effective context use as conversations grow.

6. Anthropic, How do usage and length limits work?, Claude Help Center. Accessed 30 September 2026. Used for the difference between usage allowances and conversation length, and the factors affecting consumption.

7. Anthropic, Manage costs effectively, Claude Code documentation. Accessed 30 September 2026. Used for tool requests, conversation history and caching.

8. OpenAI, Reasoning models, OpenAI API documentation. Accessed 30 September 2026. Used for reasoning tokens counting towards output usage.

It imagines a clean mechanical sequence. A model gets smarter. A worker becomes redundant. A dashboard turns green. Somewhere, a finance executive presses a tasteful grey button labelled optimise, because apparently even dystopia must comply with the design system.

That is not what the evidence suggests.

The wrong debate is whether big-tech layoffs were caused by AI or were not caused by AI. That framing is suspiciously tidy. It asks whether the robot committed the murder while ignoring the CFO beside the body holding a quarterly forecast.

A better reading is this: the first large wave of layoffs was mainly a correction for pandemic overhiring, higher interest rates, revenue pressure, and the end of free-money confidence. AI then became two things at once: a genuine productivity technology and an extremely expensive capital-allocation problem. By 2024-2026, the story had shifted from “we hired too many people” to “we need fewer people in some places because compute, infrastructure, and a smaller group of AI-critical workers now get first claim on the budget.”

That is not the same as saying AI replaced everyone. It is not especially comforting either.

The talent-hoarding hypothesis captures part of the story. Big tech has treated talent as a strategic input before. It has not always behaved like a collection of innocent employers wandering through the labour market with ergonomic keyboards and pure intentions. But the claim that the entire sector is now deliberately depressing the labour market to retain top talent is harder to establish. The incentive exists. Direct evidence of intent does not.

The useful question is not “did AI cause the layoffs?” It is this: what kinds of labour become less valuable when capital tightens, AI tools improve, and compute becomes the new headcount?

That is the question the spreadsheet is already answering, usually several meetings before anyone admits there is a spreadsheet.

The wrong comparison

Start with the comparison being quietly smuggled into the argument.

When people say “AI caused the layoffs,” they often compare AI with a fantasy version of the old labour market: stable teams, sensible hiring, accurate roadmaps, managers allocating talent rationally, and job descriptions that bore some detectable relationship to the work.

This company exists mainly in onboarding decks and executive offsites. In practice, pandemic-era hiring often resembled a grocery run before a snowstorm. Teams hired because demand was rising, competitors were hiring, capital was cheap, and the safest internal response to “can we do this faster?” was “give me twelve more people.”

The official description was strategic growth. The operational reality was sometimes an arms race with laptop stickers.

Meta’s 2022 layoff message was unusually direct about this. Zuckerberg said the company had increased investment after the pandemic accelerated online commerce, then acknowledged that he had expected the acceleration to be permanent and “got this wrong.” Meta cut more than 11,000 people, roughly 13% of its workforce. A few months later, it announced another round: around 10,000 further cuts and the closure of about 5,000 open roles under the “Year of Efficiency.”^{1 2}

That is not an AI-substitution memo. It is an overexpansion memo, a revenue-pressure memo, a “we believed our own trend line” memo. The language is corporate, but the plot requires no advanced modelling: the graph went up, everyone extended the line, and then the graph developed boundaries.

Microsoft’s January 2023 note followed a similar pattern. It announced 10,000 job cuts while saying the company would continue hiring in strategically important areas. Amazon said its reductions would exceed 18,000 roles, with Stores and PXT heavily affected. Google announced about 12,000 cuts and described them in terms of a changed economic reality and sharper prioritisation.³

Again, composition is the interesting part. These firms were not collapsing uniformly. They were cutting in one place, hiring in another, closing open roles, flattening layers, ending projects, and reallocating money. That is considerably messier than the AI-doom story, which is precisely why the AI-doom story travels better. It needs fewer tabs.

The original sin was not adopting AI. It was treating temporary demand, cheap capital, and competitive paranoia as a permanent operating model.

Then the price of maintaining the fiction increased.

Perfection was never the benchmark

The other lazy argument is whether AI is “good enough” to replace workers.

This sounds reasonable until it is compared with the systems companies already tolerate.

Most organisations do not demand perfection from their existing tools or processes. They accept broken dashboards, stale documentation, meetings in which everyone agrees and nothing changes, and roadmaps with the structural integrity of a wet croissant. Then AI appears and everyone abruptly becomes a Swiss watchmaker of epistemic purity.

The useful question is not whether AI is flawless. It is which flawed system AI is being compared with.

There is evidence that generative AI can materially improve productivity in certain workflows. In a large customer-support study, access to an AI assistant increased agent productivity, with larger gains among less-experienced workers. In a controlled software-development experiment, developers using GitHub Copilot completed a coding task substantially faster than the control group.^{4 5}

Those findings matter, but they do not mean “delete the team.” They mean the team’s shape changes.

Imagine a developer entering a legacy service called billing_sync_v2, which everyone privately understands is v4 because v3 was abandoned after the migration nobody mentions. The developer asks an AI assistant to explain the code path, identify likely duplication, draft a test, and propose a cleanup plan. The tool helps. It exposes dependencies faster than a human excavation through Slack archaeology. It also confidently overlooks the fact that a “temporary” integration from 2019 still serves one enterprise customer whose contract renewal pays for half the team’s snacks.

The AI did not eliminate engineering judgement. It relocated it.

Previously, the scarce skill was often knowing where the bodies were buried. Now it is knowing which generated explanation not to trust, which test to run, which log line matters, and which staff engineer to disturb before the cleanup becomes a revenue event.

This is where the layoff story becomes uncomfortable. If a company can produce the same output with a smaller team by cancelling low-priority projects, using AI for first drafts, automating routine coordination, and relying more heavily on senior reviewers, it may not require the same number of people. That is not science fiction. It is also not a neat one-worker-one-bot exchange.

The job does not vanish in a puff of model dust. The work is redistributed, compressed, and absorbed into the roles that remain.

Which might be funny if it were not also how burnout gets renamed leverage.

The spreadsheet is also hallucinating

Much of the AI-layoff discussion assumes companies know what their employees actually do.

That is charitable.

In many organisations, work is not described by the org chart. It is described by the person everyone contacts when the payment job fails at 11:40 p.m. The source of truth is not the architecture diagram. It is a pinned Slack message, three specialists in tribal memory, and a spreadsheet named Final_v7_REAL_final.xlsx stored inside “Q3 planning - new - approved - USE THIS.”

This matters because AI does not enter a clean system. It enters a messy one and makes the mess legible more quickly.

A product team, for example, might use AI to summarise hundreds of support tickets and customer calls. The output reports that customers are angry about onboarding. Useful. Then the team checks usage data and discovers that most of the people complaining are prospects who never activated, while retained customers are quietly struggling with permissions. The model identified the loud pain. The business needed the expensive pain.

The operating lesson is dull and therefore valuable: AI is excellent at accelerating the first pass and dangerous when the first pass is promoted into the decision.

This is also why “AI will replace managers” is usually the wrong frame. AI will replace some managerial theatre. It can summarise status updates, draft review language, cluster risks, write the first version of a reorganisation FAQ, and convert meeting sludge into something vaguely resembling decisions. Good. Some meeting sludge has enjoyed a remarkably long and undeserved career.

But the actual managerial work is deciding which trade-off the organisation is making, who is accountable, which signal is real, which team is overloaded, and which comfortable project must die. If a manager’s role consisted mainly of forwarding ambiguity between calendars, then yes, the outlook is poor. That was not management. It was packet switching with a lanyard.

AI does not eliminate the need for judgement. It removes some of the excuses for avoiding it.

That is precisely why headcount plans change.

Talent hoarding was real, just not in the tidy way

The talent-hoarding hypothesis is appealing because it explains something almost everyone in tech has witnessed.

A company hires people it does not quite know how to use. One team grows because another team grew. Requisitions become defensive weapons. Leaders collect headcount like medieval nobles collecting horses, except the horses receive RSUs and hold opinions about Kubernetes.

Beneath the comedy is a serious point: big tech has often treated talent as a scarce strategic input, not merely as a cost line.

The strongest evidence is not a collection of vibes. It is legal history. In 2010, the US Department of Justice brought a case against Adobe, Apple, Google, Intel, Intuit, and Pixar over “no cold call” agreements that restricted recruiting competition. The DOJ argued that these arrangements eliminated a significant form of competition for employees and deprived workers of information about better opportunities.⁶

That matters because it reveals the instinct directly: do not merely compete for talent; manage the market in which talent competes.

Labour economics supplies the mechanism. When workers have fewer outside options, employers gain bargaining power. Research on labour-market concentration has found that higher concentration is associated with lower posted wages. US antitrust agencies have also warned that wage-fixing and no-poach agreements can violate antitrust law and expose companies and executives to serious liability.⁷

So the broad idea is not paranoid. Labour markets are still markets. Employers still have incentives. Very large companies do not become monks merely because the cafeteria serves kombucha.

But the clean version of the talent-hoarding theory breaks when it reaches the frontier-AI labour market.

If AI scaling had made talent-blocking broadly unnecessary, the talent war at the top should have cooled. Instead, the opposite appears to have happened. Reuters reported in 2025 that top OpenAI researchers were receiving compensation packages above $10 million a year, that Google DeepMind had offered packages reaching roughly $20 million for some researchers, and that Meta had made extreme reported offers to recruit OpenAI employees. Reuters later described Meta’s AI hiring push as an intensification of Silicon Valley’s talent war.⁸

That is not de-escalation. It is an auction in a hoodie.

The better model is segmentation.

Frontier AI researchers, infrastructure specialists, and a small number of deployment leaders become more valuable. Generic surplus headcount, coordination-heavy roles, and work connected to low-priority projects become less protected. The labour market does not cool evenly. It develops weather systems.

One person receives a call about a nine-figure package. Another receives a calendar invitation from HR with no agenda. Both are participating in the same AI boom. That is the problem.

The market split in two

The most important thing AI is doing to tech labour is not simply replacing jobs. It is changing which jobs can still defend their existence.

That distinction matters for builders because “can AI perform this task?” is not identical to “should this remain a full-time role?”

Consider a routine product-operations workflow. Someone exports customer feedback, cleans a spreadsheet, tags themes, creates a deck, schedules a readout, and spends forty minutes explaining that the leading problem is “unclear onboarding,” which everyone already suspected because customers repeatedly say “the onboarding is unclear.” An AI system can compress much of that work. It can read tickets, cluster themes, generate example quotes, draft the deck, and produce a first-pass recommendation.

It can also over-weight the most verbose customers, erase meaningful edge cases, and produce recommendations that sound strategically mature because “streamline” appears twelve times.

The remaining work is verification: compare the output with usage data, revenue impact, churn, support cost, and the product surface. In a well-run team, AI reduces the time spent assembling the artefact and increases the pressure to decide. In a badly run team, AI produces a more attractive artefact so the indecision can wear a blazer.

This is the practical labour-market shift. Work dominated by assembling, reformatting, summarising, translating, routing, and light coordination becomes more vulnerable. Work involving accountable judgement, production ownership, customer trust, system design, regulatory risk, or difficult verification becomes more valuable.

Official assessments from the IMF, ILO, and OECD have been cautious about treating AI exposure as automatic job destruction. The ILO’s work argues that most occupations contain tasks requiring human input and that transformation is more likely than full automation. The OECD has similarly observed limited evidence so far of broad AI-led job loss, while still recognising high automation exposure in some occupations.⁹

That caution matters. The direction of travel matters too.

Anthropic’s own usage research has found much higher automation rates in Claude Code conversations than in general Claude.ai conversations, while its Economic Index shows AI usage heavily concentrated in computer, mathematical, and API-driven workflows.¹⁰ Challenger, Gray & Christmas reported that AI was the leading cited reason for US job cuts in April 2026. Indeed’s Hiring Lab, meanwhile, described AI-mentioned job postings as growing amid broader hiring weakness.¹¹

None of this proves that AI was the single cause of big-tech layoffs. It does explain why the latest phase feels different from 2023.

In 2023, the cleanest account was: we overhired, and money became more expensive.

By 2026, the sharper account was: we still need people, but not as many of the same people, in the same places, doing the same coordination work, while data centres and frontier teams receive the budget first.

That is colder than “the robots took the jobs.” It is less cinematic. It is also more useful.

Verification is the work

The practical response for teams is not to debate whether AI is good or bad in the abstract. That is a ritual with increasingly polished formatting.

The practical response is to map where AI changes the burden of verification.

Picture a CTO examining a 120-person engineering organisation after a budget reset. The lazy version of the exercise asks, “Which roles can AI replace?” This produces theatre. Managers defend their teams. Finance requests percentages. Someone says “10x engineer,” and the room collectively loses three IQ points.

The better workflow begins with work, not roles.

First, list the recurring workflows: incident response, customer escalation, release notes, test generation, data cleaning, roadmap synthesis, contract review, onboarding, internal tooling, and migration planning. Then ask four questions about each one.

What does AI make faster?

What does AI make easier to fake?

What verification proves the output is safe?

Who is accountable when the output is wrong?

That modest checklist accomplishes more than most AI strategy decks.

For incident response, AI may summarise logs and suggest likely causes. It may also invent a causal chain and direct the team toward the wrong service. Verification means checking metrics, traces, recent deployments, and rollback safety.

For code migration, AI may draft mechanical changes across a large codebase. It may also miss the strange customer-specific path that lacks tests because nobody funded tests when the feature launched during a sales emergency. Verification means test coverage, staged rollout, ownership, and a human who understands what “temporary exception” means in enterprise software. It generally means permanent.

For performance reviews, AI may help an EM draft clearer feedback. It may also convert vague resentment into polished prose. Verification means grounding feedback in observed behaviour, specific examples, and standards the person knew about before review season arrived like a legally compliant thundercloud.

For customer research, AI may summarise interviews quickly. It may also mistake frequency for importance. Verification means comparing themes with usage, revenue, retention, support burden, and product strategy.

The lesson is not “never use AI.” The lesson is that AI shifts effort from production towards judgement. It reduces some drafting and searching work. It increases the premium on people who know how to check, decide, own, and repair.

That is also the labour-market lesson.

If your role mainly produces artefacts that nobody verifies, you are exposed. If it owns outcomes that require verification across messy systems, it is more defensible. Preference is not performance. Busyness is not leverage. A calendar packed with coordination is not accountability, however impressive the colour coding.

The annoying question

The AI-layoff story is convenient because it allows everyone to avoid the harder conversation.

Executives can say technology changed the business. Workers can say executives used technology as cover. Commentators can choose a side and produce the required quantity of concern. Everyone receives a clean villain.

The real picture is less obliging.

Big tech overbuilt during an unusual demand shock. Higher rates and lower patience made that overbuild expensive. AI produced genuine productivity gains, genuine substitution pressure in certain tasks, and an enormous new appetite for compute. At the same time, the talent market split: hotter at the frontier, colder across broad generic tech labour, and most exposed in routine cognitive work.¹²

The talent-hoarding thesis is therefore partly correct and partly too elegant. Yes, companies have treated labour strategically. Yes, weaker outside options benefit employers. Yes, a cooler market can reduce wage pressure and quits. But proving that recent layoffs were deliberately orchestrated to depress the labour market would require evidence we do not have.

What we do have is sufficient.

We have companies closing open roles while continuing to hire in strategic areas. We have AI researchers valued like small sports franchises. We have management layers being flattened while infrastructure budgets expand. We have software teams being asked to do more with tools that are powerful, unreliable, and improving. We have junior roles under pressure because the training path was always less robust than the mythology implied.

The question is not whether AI replaces developers, PMs, analysts, recruiters, or managers.

The question is what remains worth assigning to a full-time person once the first draft is cheap, the second draft is suspicious, and the final answer still requires someone to own the consequences.

That is the annoying question for workers. It is also the annoying question for companies.

Because if the answer is “judgement,” the next question is worse.

Who, exactly, is your organisation training to exercise it?

Notes and References

1. Meta, “Mark Zuckerberg’s Message to Meta Employees,” About Meta, 2022. Used for Meta’s November 2022 layoff count, the 13% reduction, and the company’s explanation that pandemic-era acceleration had been overestimated.

2. Meta, “Update on Meta’s Year of Efficiency,” About Meta, 2023. Used for the second Meta layoff round, the planned reduction of around 10,000 employees, and the closure of about 5,000 open roles.

3. Microsoft, “Focusing on our short- and long-term opportunity,” Microsoft, 2023; Amazon, “Update from CEO Andy Jassy on role eliminations,” About Amazon, 2023; Google, “A difficult decision to set us up for the future,” Google, 2023. Used for official company explanations of 2023 layoffs and continued selective hiring or reprioritisation.

4. Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, “Generative AI at Work,” NBER Working Paper, 2023; Quarterly Journal of Economics, 2025. Used for evidence that an AI assistant increased customer-support productivity, with larger gains for less-experienced workers.

5. Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” arXiv, 2023. Used for the controlled experiment in which developers using Copilot completed a coding task 55.8% faster.

6. US Department of Justice, “Justice Department Requires Six High-Tech Companies to Stop Entering into Anticompetitive Employee Solicitation Agreements,” 2010. Used for the historical “no cold call” agreements involving Adobe, Apple, Google, Intel, Intuit, and Pixar.

7. José Azar, Ioana Marinescu, and Marshall Steinbaum, “Labor Market Concentration,” NBER, 2017 / Journal of Human Resources, 2022; Federal Trade Commission and US Department of Justice, “Antitrust Guidelines for Business Activities Affecting Workers,” 2025. Used for the relationship between labour-market concentration and lower posted wages, and for legal treatment of wage-fixing and no-poach agreements.

8. Reuters, “OpenAI, Google and xAI battle for superstar AI talent, shelling out millions,” 2025; Reuters, “Sam Altman says Meta offered $100 million bonuses to OpenAI employees,” 2025; Reuters, “Zuckerberg’s Meta Superintelligence Labs poaches top AI talent in Silicon Valley,” 2025. Used for evidence that the frontier-AI talent war intensified rather than cooled.

9. IMF, “Gen-AI: Artificial Intelligence and the Future of Work,” 2024; International Labour Organization, “Generative AI and Jobs: A global analysis of potential effects on job quantity and quality,” 2023; International Labour Organization, “Generative AI and Jobs: A Refined Global Index of Occupational Exposure,” 2025; OECD, “Employment Outlook 2023: Artificial Intelligence and the Labour Market,” 2023. Used for the distinction between AI exposure, augmentation, transformation, and displacement.

10. Anthropic, “AI’s Impact on Software Development,” 2025; Anthropic, “Anthropic Economic Index report: Economic primitives,” 2026. Used for Claude Code automation-rate evidence and concentration of AI usage in computer, mathematical, API-driven, and office-administrative workflows.

11. Challenger, Gray & Christmas, “April Job Cuts Rise 38% from March; YTD Cuts Down 50%,” 2026; Indeed Hiring Lab, “January 2026 US Labor Market Update: Jobs Mentioning AI Are Growing Amid Broader Hiring Weakness,” 2026. Used for evidence that AI is increasingly cited in layoffs while AI-mentioned job postings grow amid broader hiring weakness.

12. Reuters, “Meta CEO Zuckerberg blames layoffs on capital spending, won’t rule out more job cuts,” 2026. Used for Meta’s reported framing of layoffs as a trade-off between compute infrastructure and workforce costs during rising AI capital expenditure.

The funniest thing about the phrase “the only trustable coding benchmark” is the assumption that trust is an object you can place neatly in a leaderboard column, somewhere between Pass@1 and average token spend.

This is how engineering measurement usually goes wrong. Someone builds a useful instrument, asks it to become a religion, then acts surprised when it turns into procurement theatre with a CSV export.

DeepSWE deserves attention. It is one of the more serious public attempts to measure long-horizon coding agents on repository-level work: multi-file changes, sparse prompts, behavioural verification, regression checks, and a fixed harness. Its public repository describes 113 tasks across TypeScript, Go, Python, JavaScript, and Rust, drawn from active open-source projects. Its homepage reports model pass rates, average cost, wall-clock time, and output tokens instead of presenting one heroic score and asking it to explain the universe. [2][3]

That is good. It is not the same as being the only trustworthy benchmark for coding.

The wrong debate is whether DeepSWE is “the” benchmark. The useful question is what kind of work it forces a coding agent to do, which mistakes it can detect, and which mistakes can still stroll past the verifier wearing a lanyard.

Because coding is not one job. It is a cupboard full of jobs that engineering teams keep labelling “implementation” so the project plan remains emotionally stable.

Sometimes coding means writing a small pure function from a docstring. Sometimes it means fixing a real issue in a Python repository. Sometimes it means changing seven files without breaking the two ancient tests that only pass on the laptop nobody is allowed to reboot. Sometimes it means reading the onboarding document, discovering it describes a product that died three quarters ago, and quietly asking Darren why the payment service still depends on a cron job under his personal account.

No single benchmark captures all of that. A benchmark is a flashlight. DeepSWE is a better flashlight than many. It is not daylight.

Perfection was never the benchmark

Most benchmark arguments smuggle in the wrong comparison. They compare the benchmark with perfect knowledge, discover that it is not omniscient, and declare the entire exercise compromised.

Fine. It is flawed. So is the current system.

The current system is a staff engineer trying three coding agents on the same bug, a product manager reading a model card, a CTO staring at a leaderboard, and everyone pretending that “we should evaluate this properly” does not mean “someone should spend the next two Fridays building a private test harness while the roadmap looks at them like a disappointed parent.”

DeepSWE is useful because it improves the comparison we actually need. It does not ask only whether a model can solve toy snippets in isolation. It asks whether an agent can enter a repository, interpret a natural request, edit multiple files, and pass behavioural tests that care about what the software does rather than whether the patch uses the author’s preferred internal helper name. [3][4]

That matters because AI coding evaluation has been polluted by benchmarks that are public, old, narrow, overfit, saturated, or quietly measuring memory. HumanEval still matters for compact Python function synthesis, but it is not a proxy for changing a maintained product. MBPP still matters for entry-level Python programming problems, but nobody should select an enterprise coding agent because it can write a small list-manipulation function beneath a polite docstring. LiveCodeBench matters because it was designed around fresher contest-style coding problems and broader scenarios such as code generation, execution, test-output prediction, and self-repair. SWE-bench Verified matters historically and still provides a useful slice of GitHub issue resolution, but even OpenAI has argued that it no longer measures frontier coding progress cleanly because of contamination and test-design problems. [5][6][7][8][10]

This sounds like academic hair-splitting until you are the person approving the tool budget.

A leaderboard says Agent A is better than Agent B. In a meeting, this becomes “we have evidence.” By the time it reaches finance, it becomes “the market has validated this.” By the time it reaches engineering, it becomes “please replace the migration estimate with vibes and a screenshot.”

The serious point beneath the comedy is simple: benchmarks are not guilty because they are incomplete. They become dangerous when people forget exactly what they are incomplete about.

What DeepSWE actually tests

DeepSWE is strongest where older coding benchmarks have been visibly creaking: long-horizon, repository-level work performed under ambiguity.

According to Datacurve, the tasks are not copied from existing public pull requests. They are authored as original tasks against active open-source repositories. The reference solution is held out. The prompt is what the agent sees. The verifier is executable. The intended target is observable behaviour, not a secret implementation shape. [3][4]

That design choice matters more than the score.

Imagine an engineer using an AI agent to add cancellation support to a JavaScript engine. The weak version of the task is: “change this exact function, add this exact guard, return this exact value.” The stronger version is: “make cancellation stop all queued work associated with an evaluation without breaking unrelated jobs.” The first measures patch obedience. The second begins to measure engineering.

Datacurve’s qualitative audit includes an example in this shape: an agent mostly implemented cancellation support but missed an important path where Promise callbacks could still be queued after cancellation. That is precisely the kind of failure a shallow benchmark can miss and a behaviour-first repository task is more likely to expose. [4]

This is why DeepSWE is interesting. It is not merely harder. Harder is easy. You can make any benchmark harder by writing a prompt that sounds as though it was produced by a compliance department and a dungeon master trapped in the same elevator.

The useful difference is what DeepSWE accepts as evidence. It rewards an agent for reading the repository, preserving behaviour, interpreting a shorter request, making a coherent multi-file change, and declining to treat the tests as decorative furniture.

That is close to how teams want coding agents to behave. Not magic. Not autonomous product ownership. Just a competent assistant that can enter a codebase without immediately deciding that abstractions are optional and every edge case is a lifestyle choice.

The current system is also hallucinating

The phrase “AI hallucination” has become extremely convenient. It makes the model sound uniquely irrational while the surrounding organisation stands nearby holding three inconsistent Jira tickets and a Confluence page last updated during the previous monarchy.

Benchmarks hallucinate too. They simply do it with more formal notation.

They hallucinate when a test suite checks one implementation detail and calls it correctness. They hallucinate when a public task rewards a model for having encountered the answer before. They hallucinate when a prompt tells agents not to write tests and the leaderboard later acts surprised that test-writing behaviour disappears. They hallucinate when a single pass rate quietly conceals cost, time, token spend, and review burden.

DeepSWE attempts to reduce several of these failure modes. Its maintainers report original tasks, programmatic verifiers, regression checks, and a fixed mini-swe-agent harness. They also report a verifier audit in which DeepSWE had a 0.3% false-positive rate and a 1.1% false-negative rate on the reviewed sample, compared with 8.5% and 24.0% for SWE-Bench Pro. That is a striking claim. It is also a maintainer-reported claim using an LLM-based analyser as part of the audit, so the appropriate reaction is not worship. It is “excellent, now replicate it.” [4]

This is the adult position, which is why it performs poorly in rooms that would prefer a winner.

DeepSWE may be a better measurement tool for one important construct. It may expose distinctions that older leaderboards compress. The public page cited here reports a substantial spread between model configurations, with GPT-5.5 [xhigh] at 70% +/- 4% Pass@1 and several other frontier configurations materially lower. That is useful signal, particularly compared with benchmarks where top models have converged into one polite blob. [2]

But signal is not truth. Signal is evidence produced under a measurement design.

A thermometer can tell you the temperature of a room. It cannot tell you whether the building has poor insulation, whether finance approved the heating bill, or whether Darren opened the window because the server closet smells like hot plastic. This is not a criticism of thermometers. It is a criticism of people who use one reading to explain the entire office.

Verification is the work

The operating lesson for builders is not “use DeepSWE” or “ignore DeepSWE.” It is that verification has become the real work.

Consider a normal product team evaluating a coding agent for repository maintenance. The demonstration is impressive. The agent changes three files, explains itself clearly, and produces a summary with the confidence of someone who has never had to be on-call. The team runs the tests. Most pass. The agent apologises for the one that fails and proposes a patch. Everyone has a small spiritual experience.

Then the staff engineer asks three dull questions, which are generally the important ones:

Did it change the right behaviour, or just satisfy the visible tests?

Did it preserve unrelated behaviour, or only avoid breaking the happy path?

Did it reduce future maintenance, or did it create a clever little shrine to today’s prompt?

This is where DeepSWE becomes useful as a pattern, not merely as a leaderboard. Behavioural verifiers, regression checks, task-specific tests, cost reporting, time reporting, and trajectory audits are not benchmark luxuries. They are increasingly the basic equipment of serious AI-assisted engineering inside a company.

The workflow is boring and therefore has a suspicious chance of being correct. Pick real internal tasks. Freeze the base commit. Give the agent the same prompt a developer would actually write. Let it operate in the same harness you intend to use. Run the existing tests, targeted behavioural tests, static analysis, security checks, and review. Track pass rate, review time, rework, cost, latency, rollback risk, and the specific failure mode.

Then perform the annoying part: read the failures.

The failures are the product. A model that misses one parallel requirement is different from a model that breaks integration. A model that writes tests and catches its own mistake is different from one that produces a majestic patch and disappears before the bill arrives. A model that succeeds only when the answer is present in git history is not solving your problem. It is doing archaeology with an API key.

This is why a private evaluation suite matters. DeepSWE can tell you something about frontier-agent capability on open-source repository tasks. It cannot tell you whether the agent understands your strangler-fig migration, feature-flag conventions, audit-trail requirements, or the part of the billing service that everyone calls “temporary” in the same tone archaeologists use for pottery shards.

The product wrapper still matters

One of DeepSWE’s best choices is also one of its limitations: it fixes the harness.

Datacurve runs models through mini-swe-agent for consistency. That reduces scaffolding noise. The leaderboard is attempting to compare model capability rather than the hidden advantages of one vendor’s product environment, prompt, editing tool, or carefully massaged workflow. [2][4]

This is clean. In practice, nobody buys “model capability” in a jar.

Teams use Claude Code, Codex CLI, Cursor, Gemini CLI, internal wrappers, security scanners, branch protection, code-review bots, ticket systems, and whatever script Michael wrote during the incident because it was 2 a.m. and “temporary” had once again become architecture.

A fixed harness answers one question: how do these models behave when they are forced through the same narrow pipe?

A native product answers another: how does the complete system perform when developers actually use it?

Both questions matter. Confusing them is how teams buy a model because it wins a benchmark and then discover that generation was not the expensive part. The expensive parts are integration, review, permissions, context packaging, security approval, developer trust, and the small issue of whether the thing can open the correct files without turning the repository into a choose-your-own-adventure novel.

This is also why preference is not performance. RealHumanEval found that benchmark improvements can translate into programmer productivity gains, but not proportionally. It also found that programmer preferences did not necessarily track actual performance. [9]

That should bother anyone who relies heavily on demonstrations. Demos are preference machines. They are designed to make the tool feel obvious. Production is a performance machine. It is designed to punish every missing edge case you applauded earlier.

What teams should actually do

Use DeepSWE as a serious external signal. Do not use it as a solitary purchasing ritual.

For researchers, it belongs near the centre of the long-horizon coding-agent evaluation stack. Pair it with function-level benchmarks, freshness-oriented benchmarks, and quality-oriented evaluations covering security, maintainability, and efficiency. Report the harness, model version, effort mode, cost, time, token use, and uncertainty. Without those details, the score is not a result. It is a badge.

For practitioners, the minimum sensible evaluation resembles a release checklist more than a leaderboard screenshot.

Begin with the public evidence: DeepSWE for long-horizon repository work; HumanEval or MBPP for small synthesis primitives; LiveCodeBench for fresher algorithmic coding; SWE-bench-style tasks for issue resolution; and human-in-the-loop studies or internal pilots for productivity. Then add your private work: five to twenty representative repository tasks, held-out behavioural tests, review by your engineers, and explicit measurement of how much effort the agent saves or merely relocates downstream.

Downstream is where lies go to become tickets.

Do not ask only “did the agent pass?” Ask who had to verify it, what they checked, what they missed, how long the checking took, and whether the resulting code is something you would willingly maintain after the agent has wandered off to generate a haiku about root cause analysis.

The uncomfortable conclusion is that good AI coding evaluation looks less like model ranking and more like engineering management. It requires boring infrastructure, realistic tasks, clear ownership, review discipline, and sufficient humility to admit when a tool is useful but not safe unattended.

That is not a strategy-deck sentence. That is the work.

The annoying question

DeepSWE is not the only trustable coding benchmark. That claim is too strong and, more importantly, too small.

The better claim is sharper: DeepSWE is one of the most useful current public benchmarks for long-horizon, behaviour-verified, repository-level coding agents. It measures a slice of work that teams genuinely care about and that older leaderboards frequently blur. It exposes failures that resemble real engineering failures: missed requirements, integration errors, regressions, weak verification, and suspicious assistance from public history.

But it does not measure coding in general. It does not fully measure native product ergonomics, enterprise context, security, maintainability, review quality, behaviour on private codebases, or whether a developer actually gets their afternoon back. It does not convert judgment into a number. It gives judgment something better to inspect.

This is the useful frame.

The point of DeepSWE is not that we have finally discovered the one true benchmark. The point is that a serious benchmark makes the rest of our evaluation habits look lazy.

That is why the annoying question for teams is not “Which model won?”

It is: “What would our own benchmark reveal about the way we already ship software?”

Most organisations will not enjoy the answer. Which is usually how you know the benchmark is beginning to work.

Notes and References

1. Uploaded report, “DeepSWE and the Trustworthiness of Coding Benchmarks,” 2026. Used for the source framing that DeepSWE is a strong current public benchmark for long-horizon repository-level coding-agent evaluation, but not the only trustworthy benchmark for all coding work.

2. Datacurve, “DeepSWE,” Datacurve, 2026. Used for the current public leaderboard snapshot, including the 30 May 2026 update, Pass@1, cost, time, output-token fields, and the statement that all models are run on mini-swe-agent for consistency.

3. Datacurve, “datacurve-ai/deep-swe,” GitHub, 2026. Used for the public repository description: 113 tasks across TypeScript, Go, Python, JavaScript, and Rust; task format; held-out reference solution; and program-based behavioural verifiers.

4. Datacurve, “DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks,” Datacurve Blog, 2026. Used for task-construction methodology, original-task claims, verifier design, false-positive and false-negative audit numbers, harness discussion, qualitative failure taxonomy, and examples of missed requirements and test-writing behaviour.

5. SWE-bench, “SWE-bench Verified,” SWE-bench, 2026. Used for the claim that SWE-bench Verified is a human-filtered subset of 500 SWE-bench instances for evaluating coding agents and language models.

6. OpenAI, “HumanEval: Hand-Written Evaluation Set,” GitHub, 2021. Used for HumanEval as a function-level code-generation benchmark and evaluation harness associated with “Evaluating Large Language Models Trained on Code.”

7. Google Research, “Mostly Basic Python Problems Dataset,” GitHub, 2021. Used for MBPP as around 1,000 crowd-sourced entry-level Python programming problems with task descriptions, solutions, and automated tests.

8. Naman Jain et al., “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code,” arXiv, 2024. Used for LiveCodeBench as a continuously updated coding benchmark with scenarios beyond code generation, including self-repair, code execution, and test-output prediction.

9. Hussein Mozannar et al., “The RealHumanEval: Evaluating Large Language Models’ Abilities to Support Programmers,” OpenReview, 2024. Used for the human-in-the-loop finding that benchmark gains can improve productivity but not proportionally, and that programmer preference signals do not necessarily correlate with actual performance.

10. OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities,” OpenAI, 2026. Used for the contamination and test-design concerns around SWE-bench Verified and the broader argument that public coding benchmarks can stop measuring genuine frontier progress cleanly.

An open model lands near the top of a public leaderboard. Someone posts a chart. Five seconds later, the verdict arrives: the frontier labs are cooked.

I think that conclusion is too tidy. It confuses a model with a product, then quietly confuses the product with the business built around it.

The wrong debate is whether an open model can match a closed model on a public test. Of course it can, in particular slices and sometimes very impressively. The useful question is what else must be copied before a customer can move serious work without caring who made the underlying model.

My answer is simple: the moat has moved up the stack.

Benchmark parity still matters. It pressures prices. It embarrasses marketing decks. It makes lazy product managers nervous, which remains a public service. But it is no longer the cleanest map of competitive power. A high-value AI product is a runtime system wrapped around a model: tool orchestration, retrieval, policy enforcement, private evaluations, monitoring, enterprise controls, distribution, pricing, and trust.

You can scrape answers. You can fine-tune a student. You can copy the vibes. What you cannot cheaply scrape is the machinery that makes the thing work on Monday morning inside a company full of permissions, angry engineers, compliance reviews, and audit logs.

* * *

Benchmark parity is a bad trophy

The 2026 AI Index gives us the more boring and therefore more useful version of the story. It reports that top-model performance is converging, that the leaders are packed more tightly together, and that competition is shifting towards cost, reliability, and domain-specific performance. It also reports that the open-versus-closed gap, after narrowing in 2024, widened again, with the top closed model ahead of the top open model by 3.3 percentage points as of March 2026. [1]

That is not a victory lap for closed models. It is a warning against treating one chart as sacred scripture with coloured bars.

Public benchmarks are useful instruments, but they are increasingly noisy ones. Stanford’s report points to rapid benchmark saturation, invalid-question rates as high as 42 percent on some widely used evaluations, and concerns that leaderboard standing can partly reflect adaptation to the platform rather than general capability. [1]

This sounds obvious, but teams miss it with impressive consistency: when the test is unstable, parity on the test is not a durable economic fact.

LiveBench exists because static benchmarks are vulnerable to contamination and obsolescence. Its designers release new questions regularly and use objective ground-truth answers to reduce dependence on LLM judges. [2] Humanity’s Last Exam exists for a similar reason: older academic tests were becoming too easy for the strongest systems, so the measurement target had to move. [23]

The deeper problem is that benchmark capability is jagged. A model can look equivalent on a mathematics set and then behave quite differently on long-context retrieval, tool use, refusal calibration, latency, consistency, or enterprise permissions. Lost in the Middle showed that language models can perform worse when relevant information appears in the middle of a long context, even when the same information is easier to use at the beginning or end. [3] That sort of failure rarely appears in the leaderboard screenshot being passed around social media like a medical certificate.

Preference is not performance. A public arena vote is not the same thing as a production system surviving a month of messy workflows.

The product is not the model

The easiest way to misunderstand the frontier labs is to keep imagining that they sell one static model in a glass box. That was closer to true during the early chatbot phase. It is much less true now.

OpenAI’s Responses API is a useful example. The official documentation describes a stateful interface for agent-like applications, with built-in tools including web search, file search, computer use, Code Interpreter, remote MCP support, and reasoning summaries. [4] [5] A leaderboard score does not recreate that product surface. Copying the answer style does not reproduce the server-side tool loop, preserved context, encrypted reasoning items, or enterprise controls surrounding the API.

Google’s Gemini stack is similarly more than a model endpoint. The developer documentation covers grounding with Google Search, context caching, structured outputs, built-in tool combinations, and file or function calling. [11] [12] Google’s Gemini 3.1 Pro model card describes internal safety evaluations, human red teaming, and continuous testing under its Frontier Safety Framework. [14] Again, the interesting thing is not only the model. It is the model’s placement inside search, cloud, developer tools, enterprise surfaces, and a safety-release process.

Anthropic’s moat is different again. It is not merely that Claude answers well. Anthropic has built a safety- and developer-oriented deployment regime around tool use, prompt caching, web search, system-card discipline, and usage restrictions. [7] [8] It also now discusses detecting industrial-scale distillation attempts through account behaviour, request metadata, traffic patterns, and capability-targeted prompts. [10]

Meta is the useful counterexample. Meta deliberately weakens the “weights secrecy” moat by releasing Llama models openly. That does not mean Meta has no moat. Its strategy is ecosystem capture: developer mindshare, distribution, open weights, safety tools, partners, and the ability to keep reinvesting at enormous scale. Meta’s own Llama 3.1 announcement says its licence changes allow developers to use outputs from Llama models, including the 405B model, to improve other models. [15] That is not a lab forgetting to defend itself. It is a lab choosing a different layer to defend.

So when someone says, “the open model caught up,” the first response should be: caught up to what? The static score? The assistant product? The agent runtime? The compliance posture? The distribution channel? The cost curve?

The phrase “model moat” is now too small. The product moat is the thing.

Distillation is real. It is just narrower than the panic

The anti-hype version of this argument is not that extraction is imaginary. It is very real.

The model-stealing literature goes back years. Tramèr and colleagues showed in 2016 that black-box prediction APIs could be copied with surprising fidelity in several supervised-learning settings. [17] Later work such as Knockoff Nets showed that attackers could train functional substitutes for vision models by querying a target and using its responses as labels. [18] Wallace and colleagues showed that commercial machine-translation systems could be imitated through query access, while Xu and colleagues showed that NLP API imitators could sometimes outperform the target on transferred domains. [19] [20]

Those papers matter because they kill the comforting story that an API boundary is automatically a wall. It is not. An API can also be a teacher, albeit one that sends invoices.

Recent LLM work makes the concern sharper. LoRD studies locality-reinforced extraction for aligned LLMs and argues that alignment-aware extraction can be more query-efficient than naive imitation. [21] A 2026 trace-inversion paper goes further: it reports that a student model fine-tuned on synthetic reasoning traces inferred from a commercial model’s answers and summaries improved substantially on MATH500 and JEEBench compared with fine-tuning on answers and summaries alone. [22]

That is not nothing. It is also not instant cloning by cinematic progress bar.

The same research is revealing about the limits. These attacks are usually task-targeted. They depend on query budgets, public corpora, surrogate models, filtering, relabelling, and offline fine-tuning. They transfer slices of behaviour. They do not copy the teacher’s full internal training mixture, hidden routing, safety monitors, enterprise permission model, tool environment, private launch gates, or customer trust surface.

Anthropic’s February 2026 post is important here, but it should be read carefully. Anthropic says it detected industrial-scale campaigns by DeepSeek, Moonshot AI, and MiniMax involving millions of exchanges through fraudulent accounts, with prompts targeting reasoning, coding, tool use, and agentic behaviour. [10] That is a serious provider-side claim. It is evidence that labs treat capability siphoning as operationally real. It is not independent proof that any one actor obtained complete product equivalence.

The realistic threat is not a film scene in which someone steals the entire frontier model overnight. The realistic threat is continuous siphoning: cheaper followers copying selected behaviours, compressing commodity margins, and shortening the distance between leader and fast follower in specific domains.

That is dangerous enough. We do not need to give it dramatic lighting.

Where the moat actually moved

First, inference-time systems. The best products now do more than answer text. They call tools, retrieve files, cache context, ground claims, execute code, remember workflow state, and route around failure. The visible output is the final inch of the system, not the system itself.

Second, private evaluation and release gates. Public benchmarks tell the world what everyone can see. Serious deployment depends on tests customers never see: regression suites, abuse probes, internal red-team cases, latency targets, tool-use evaluations, refusal calibration, long-context retrieval checks, and domain-specific failure modes. A scraper can sample outputs. It cannot easily infer the hidden evaluation set that shaped the release decision.

Third, trust and liability. Enterprises do not buy only token prediction. They buy contractual posture, privacy defaults, account controls, admin consoles, SSO, retention settings, auditability, support, incident handling, and somebody to blame when things break. These are boring moats. Boring moats are excellent because competitors keep walking past them in search of something photogenic.

Fourth, distribution. Google has Search, Workspace-adjacent surfaces, Vertex AI, AI Studio, and developer channels. OpenAI has consumer distribution, API adoption, and an agent platform. Anthropic has a strong developer and enterprise wedge around Claude. Meta has the open ecosystem. These are not identical strategies, which is precisely the point. Competition has fragmented across layers.

Fifth, legal friction. Terms of service are not magical shields, but they raise the cost of extraction conducted openly. OpenAI, Google, and Anthropic all restrict using their services or outputs to build competing models, or to reverse engineer or extract the underlying system. [6] [9] [13] A determined attacker can violate terms. A large commercial buyer cannot comfortably build its AI strategy on violating them and hope procurement does not notice.

This is why the phrase “open-source killed the moat” is too blunt. Open models absolutely compress the commodity layer. They make generic summarisation, basic coding assistance, formatting, translation, and many benchmark-facing demonstrations cheaper. They also force incumbents to justify premium pricing with reliability and integration rather than mystique.

But the moat did not disappear. It moved from the static artefact to the operating system around the artefact.

A concrete workflow: the enterprise coding assistant

Take a suitably boring example: a company wants an internal coding assistant for a large, ugly codebase. This is not a demonstration repository. It has legacy services, half-migrated libraries, secret configuration patterns, flaky tests, permission boundaries, and several sacred files nobody should touch unless the staff engineer has had coffee.

A distiller can query the public model with coding tasks and collect good answers. That helps. It may produce a cheaper model that writes decent patches for common problems. It may even learn some of the teacher’s style: clearer explanations, better decomposition, and more cautious refactoring.

But the production assistant must do other things.

It must retrieve the right files, not merely plausible ones. It must respect permissions. It must know when a failing test is a real signal and when the test suite is lying because a fixture was written during a previous geological era. It must call a code interpreter or build system. It must preserve context across a multi-step edit. It must explain changes in the format the team actually reviews. It must avoid leaking secrets into logs. It must produce traces that compliance and security teams can inspect. It must be monitored for abuse, prompt injection, data exfiltration, and suspicious usage patterns.

The answer text is the visible residue of that workflow. It is not the workflow.

This is the practical distinction that keeps disappearing. If the job is “write a quick helper function,” model parity matters greatly. If the job is “operate safely inside our production engineering system,” the model is only one component. You still need connectors, permissions, retrieval, evaluations, logging, rollout controls, and trust.

That is why open models can be both extremely important and insufficient to erase the frontier labs’ advantage. Both things can be true. Annoying, I know.

The boring prediction

My forecast is a barbell.

At one end, open models and fast followers continue commoditising broad-purpose inference. The price of generic text generation falls. The quality of local and self-hosted models improves. More companies decide that good enough is not merely good enough, but preferable because it is cheaper, more controllable, and closer to their data boundary.

At the other end, frontier labs push further into high-trust, high-integration, high-liability surfaces: coding agents, enterprise workflows, search-grounded systems, multimodal orchestration, regulated deployments, private-evaluation advantage, and products where failures are expensive enough that customers pay for the surrounding machinery.

Distillation accelerates the first end of the barbell. It narrows the commodity layer and makes fast-following easier. It also encourages providers to expose less reasoning signal, watermark more, fingerprint more traffic, verify more accounts, and separate public demonstration behaviour from high-value production behaviour. That is the natural defensive response when the attack is not theft of raw weights, but capability siphoning through outputs.

The stronger labs should not be complacent. Open models will keep cutting into margins. Customers will become better at asking why a premium model deserves the premium. Some products that appear defensible today will become wrapper dust tomorrow.

But the stronger critics should also be more precise. “The open model matched a benchmark” is not the same statement as “the frontier product has no moat.” It is a claim about one layer, often the most visible layer, and visibility is not the same thing as importance.

What actually matters is whether the hard-to-copy parts of the system sit where customers feel the pain.

So the useful question is not whether open models will catch frontier labs on more charts. They will. The useful question is whether you are buying intelligence, or buying the machinery that makes intelligence useful after the chart is gone and the work begins.

* * *

Notes and References

1. Stanford Institute for Human-Centered AI, 2026 AI Index Report, Technical Performance chapter. https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance

2. Colin White et al., LiveBench: A Challenging, Contamination-Free LLM Benchmark, ICLR 2025 / LiveBench project. https://github.com/LiveBench/LiveBench

3. Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics, 2024. https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00638/119630/Lost-in-the-Middle-How-Language-Models-Use-Long

4. OpenAI, New tools and features in the Responses API, 2025. https://openai.com/index/new-tools-and-features-in-the-responses-api/

5. OpenAI API documentation, Responses API overview and migration guidance. https://developers.openai.com/api/docs/guides/migrate-to-responses

6. OpenAI, Terms of Use, restrictions on reverse engineering, extraction, rate-limit bypass, and using output to develop competing models. https://openai.com/policies/row-terms-of-use/

7. Anthropic documentation, Prompt caching. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

8. Anthropic API release notes, tool use, web search, and platform features. https://docs.anthropic.com/en/release-notes/api

9. Anthropic Help Center, Can I use my Outputs to train an AI model?, March 2026. https://support.claude.com/en/articles/12326764-can-i-use-my-outputs-to-train-an-ai-model

10. Anthropic, Detecting and preventing distillation attacks, February 2026. https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks

11. Google AI for Developers, Grounding with Google Search. https://ai.google.dev/gemini-api/docs/google-search

12. Google AI for Developers, Context caching. https://ai.google.dev/gemini-api/docs/caching

13. Google AI for Developers, Gemini API Additional Terms of Service, effective March 23, 2026. https://ai.google.dev/gemini-api/terms

14. Google DeepMind, Gemini 3.1 Pro Model Card, February 2026. https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card.pdf

15. Meta, Introducing Llama 3.1: Our most capable models to date, July 2024. https://ai.meta.com/blog/meta-llama-3-1/

16. The Llama Team, The Llama 3 Herd of Models, 2024. https://arxiv.org/abs/2407.21783

17. Florian Tramèr et al., Stealing Machine Learning Models via Prediction APIs, USENIX Security, 2016. https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer

18. Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz, Knockoff Nets: Stealing Functionality of Black-Box Models, CVPR 2019. https://openaccess.thecvf.com/content_CVPR_2019/html/Orekondy_Knockoff_Nets_Stealing_Functionality_of_Black-Box_Models_CVPR_2019_paper.html

19. Eric Wallace et al., Imitation Attacks and Defenses for Black-box Machine Translation Systems, EMNLP 2020. https://aclanthology.org/2020.emnlp-main.446/

20. Yifei Xu et al., Student Surpasses Teacher: Imitation Attacks for Black-Box NLP APIs, COLING 2022. https://aclanthology.org/2022.coling-1.252/

21. Zi Liang et al., Yes, My LoRD: Guiding Language Model Extraction with Locality Reinforced Distillation, 2024. https://arxiv.org/abs/2409.02718

22. Tingwei Zhang, John X. Morris, and Vitaly Shmatikov, How to Steal Reasoning Without Reasoning Traces, arXiv, March 2026. https://arxiv.org/abs/2603.07267

23. Long Phan et al., Humanity’s Last Exam, arXiv, 2025/2026. https://arxiv.org/abs/2501.14249

The more dangerous promotion is giving an extremely competent person a new title and assuming the job underneath has changed to match.

Often, it has not. Not enough.

Their calendar still rewards rescue. Their status still comes from having the best answer. Their manager still wants them to keep the hard parts close. Their team still sends uncertainty upstairs because that is where certainty lives.

Six months later, everyone is surprised that the new leader is exhausted, the team is dependent, and the bench is thinner than before. Apparently, the title was supposed to handle the transition.

I think this is one of the quietest traps in leadership development. High performance is not the problem. Unedited performance is.

Speed, precision, ownership, personal reliability: these behaviours can make someone exceptional in a defined individual-contributor role. They can also become constraints when the job starts creating value through other people’s capability, coordination, judgement, and growth.

Preference is not performance. Output is not potential. Being the person everyone trusts with the hardest work is not the same as making everyone else better at hard work.

The wrong debate is whether high performers make bad leaders

They do not, as a category. That framing is lazy.

High performers often bring useful raw material: credibility, discipline, pattern recognition, stamina, taste, and enough scars to recognise good work. In technical teams, that credibility matters. Engineers can smell fake authority through concrete.

The useful question is not whether high performers are leadership risks. It is what kind of performance the organisation keeps rewarding after it promotes them.

Leave personal throughput on the scoreboard and the new manager will keep playing the old game. Not because they are vain. Because the instructions are perfectly clear.

The research points to this selection problem. Firms often promote people for visible performance in their current role, even though management effectiveness depends on partially different signals. The classic Peter Principle problem is not that workers become stupid after promotion. It is that the selection system gave too much weight to the wrong evidence for the next job.

So the star sales rep becomes the manager who still wants to close the largest deals personally. The best engineer becomes the tech lead who quietly rewrites every design doc. The reliable operator becomes the supervisor for whom every escalation is a referendum on their personal standards.

They have not stopped caring. They are still caring through the old mechanism.

The old scoreboard follows them upstairs

The first managerial transition is an identity shock disguised as a promotion.

Before, the question was: Can you solve the problem? Afterwards, it becomes: Can the team solve more and better problems because of you?

That is a different production system, not a more senior setting on the old one.

The high performer wants to be useful. The new leader has to learn a less immediately satisfying version of usefulness. Less rescue, more context. Fewer answers, more judgement transfer. Less personal polish, more team-owned quality.

This sounds obvious until the work arrives. Finishing a task feels productive. Coaching someone through it feels slow. Rewriting a proposal feels safe. Letting someone else ship a merely good version feels suspiciously like negligence.

The organisation often helps the trap along. It promotes the expert, leaves them with half their old workload, widens their span, and tells them to be more strategic. This is not a transition plan. It is two jobs wearing one title.

Recent workplace data sharpens the problem. Larger spans of control are becoming more common in some organisations, while managers often still carry substantial individual-contributor work. Under those conditions, execution fixation is not merely a habit. It is a rational response to a badly defined job.

The newly promoted high performer responds with the moves that have always worked. They absorb ambiguity. They carry quality. They become the final reviewer, the escalation path, and the shadow owner of every important decision.

For a while, the team gets faster. Then it gets weaker.

The first symptom is that they help too much

The giveaway is, ‘I can do it faster.’

Usually, they can. That is what makes the trap work.

Ask only who can finish this task fastest today and the high performer should keep doing it. But leadership is not optimised for today’s isolated task. It is optimised for the team’s future capacity to handle a class of tasks without heroic intervention.

This is where helpfulness starts doing damage.

The star engineer who jumps into every code review improves the code and teaches the team that quality lives in her head. The finance lead who rewrites every board deck protects standards and teaches the team that only formatting has been delegated, not judgement. The product lead who approves every small decision stays close to the work and makes speed dependent on his attention span.

High standards are not the enemy. Standards are useful. Taste is useful. A leader with no quality bar is just a calendar with opinions.

The trouble starts when the standard and the leader’s preferred method become inseparable. Nobody else gets to own meaningful work. Perfectionism becomes expensive here: what looks like care from the leader’s side can feel like infantilisation from the team’s.

The bill arrives late. That is why organisations tolerate the arrangement for so long.

At first, the team looks protected. Deadlines are saved. Quality holds. Senior stakeholders relax because the old hero is still within reach. Then reviews take longer. Decisions queue. The best people leave for places where they can make calls. Junior people learn compliance instead of judgement. Succession plans become fiction with better typography.

The manager has not become lazy. The system has become leader-dependent.

Delegation is not a moral virtue. It is an operating system.

We discuss delegation as though it were mostly a personality trait. Some leaders ‘let go’. Others ‘struggle to let go’. Fine. That still leaves rather a lot of operating detail unresolved.

Delegation fails when authority, quality, risk, and feedback are vague. A high performer will not release control into fog. Nor should they.

The practical answer is to make delegation boringly explicit.

First, name the decision rights. Who decides? Who advises? Who needs to be informed? Which decisions require no approval? Which need a second pair of eyes because the risk is genuinely high?

Second, define the quality bar. Not ‘make it good’. That tells nobody anything useful. Good compared with what? Which trade-offs are acceptable? What must never break? Where can style vary? Where is accuracy non-negotiable?

Third, define the feedback loop. Does the leader inspect the work before the decision, afterwards, or only when an exception is triggered? How will the learning be captured so the next decision needs less oversight?

That is the difference between delegation and abandonment. Dropping work on someone and waiting for maturity to appear is not delegation. Transferring ownership with enough context, constraints, and a review rhythm for judgement to grow is.

More direct control can be legitimate in novice teams, safety-critical work, regulated contexts, or crisis conditions. Control itself is not the problem. Control that never reduces as capability rises is.

A good leader should be able to distinguish a risk tier from a comfort blanket.

A concrete reset for the star who just became the boss

Imagine Priya, a senior engineer promoted to lead a platform team.

Before the promotion, she was the person you wanted on the worst incident and the ugliest migration. She could see around corners. Her design docs were crisp. Her code reviews were annoying in the correct way. Naturally, she got promoted.

Three months later, the team is busy but somehow smaller than its headcount. Every design waits for Priya. Every production issue pings Priya. The strongest engineer has stopped proposing changes because Priya will improve the proposal anyway. The juniors are learning her preferences faster than they are learning engineering judgement.

Telling Priya to care less would be silly. The task is to change where her care goes.

Week one: audit the calendar. Separate personal production, review and approval, coaching, hiring and staffing, stakeholder alignment, and strategic design. If personal production still dominates, the promotion has not really happened. The pay band has changed clothes.

Week two: build a decision ladder with the team. Level one: bring me the facts and I decide. Level two: bring me a recommendation and I approve. Level three: decide, inform me, and capture the reasoning. Level four: decide without me unless an exception is triggered. This is not a race to put everything at level four. It is a way to make movement visible.

Week three: transfer one genuinely important work package, not just the administrative crumbs. Delegating only low-stakes scraps teaches the team that trust is decorative. Choose something material, define the quality criteria, agree on check-in points, and let the owner choose the method unless that method creates real risk.

Weeks four to eight: replace rescue with coaching. In one-to-ones, Priya stops asking only, ‘What is blocked?’ and starts asking, ‘What decision are you trying to make? What options did you reject? What evidence would change your mind? Where do you want me to be precise, and where do you want me to stay out?’

Month three: measure whether dependence has fallen. Not by vibes. Track review latency, decision escalations, rework rate, time spent in personal production, and whether at least one direct report can now handle a class of decisions that previously required Priya.

None of this is glamorous. That is why it works.

Priya keeps her standards. The team gains access to the judgement behind them. The organisation gets a leader rather than a heroic bottleneck with a nicer title.

What to promote for instead

Promotability should stop being a victory lap for individual output.

Output matters. It should. But when the next role creates value through others, the assessment has to move towards leadership-relevant behaviour.

That means asking different questions.

How has this person improved someone else’s judgement in the last year? What can their team now do without them? When have they handed over an important task and accepted a result that met the standard without following their preferred method? What feedback about their impact has been hardest to hear? Do people bring them weak signals early, or only polished answers late? Who gets stronger after working with them?

These are harder to score than revenue, code output, tickets closed, utilisation, or delivery heroics. That is the point. The next job does not become visible just because you stare harder at the old dashboard.

A better promotion process uses several lenses: structured behavioural interviews, work simulations, multi-rater feedback, evidence of coaching behaviour, and stretch assignments before the formal move. The aim is not to find a flawless person. It is to find someone who can learn the new game without letting their old strengths automatically sabotage it.

This also requires a respected specialist path. Many organisations quietly push excellent specialists into management because that is where status, compensation, and influence live. Then they are surprised when some of those people continue behaving like specialists. The org chart made the deal before the person did.

Dual ladders are not a kindness. They are risk management. They reduce the pressure to turn every master craftsperson into a people manager, then spend two years repairing the predictable damage.

The manager’s manager is part of the problem

A newly promoted high performer rarely makes this change alone.

Their manager has to stop buying the old product. Many transitions quietly die here.

The senior leader says, ‘Delegate more,’ then privately routes the hardest task back to the star because the deadline matters. They say, ‘Build the team,’ then save their loudest praise for the manager who personally rescues the release. They say, ‘Be strategic,’ then leave the entire old production load in place.

People are not confused by the speeches. Their incentive-reading skills are excellent.

For the high performer to become a leader, their manager has to change the contract. Team capability must become the work, not a pleasant extra to fit in after the real work is finished.

That contract needs an explicit reduction in individual-contributor load, a definition of decisions the team should make without the leader, a small set of coaching expectations, and bench strength included in performance reviews. Otherwise, leadership development is theatre: everyone nods at the workshop, then returns to the same operating system.

The problem is usually not that the high performer refuses to grow. It is that the organisation keeps paying them in prestige for staying exactly where they are.

The useful question

High performance should still matter. I do not want managers who have never done hard things. I do not want leaders whose main skill is narrating complexity from a safe distance. Technical respect, commercial judgement, operational discipline: these are assets.

But leadership potential is not current performance with the volume turned up.

The useful question is not, ‘Who is the best?’ It is, ‘Best at what production system?’

When the next role requires output through others, the promotion question changes. What happens to the system when this person gets more authority? Does work multiply through them, or queue behind them? Do people grow sharper around them, or more cautious? Does the team inherit their judgement, or just wait for their approval?

A high performer who learns to transfer judgement is dangerous in the best way. They bring standards without hoarding ownership, taste without cloning themselves, and urgency without turning everyone else into spectators.

A high performer who cannot make that shift becomes something else. Not a bad person. Not a failed talent. A ceiling with a track record.

So before promoting the person who saves the day, ask the slightly annoying question.

After they get the title, will there be more people who can save the next one?

* * *

Notes and References

1. Uploaded research brief: When High Performance Constrains Leadership Potential, May 2026.

2. Alan Benson, Danielle Li, and Kelly Shue, ‘Promotions and the Peter Principle,’ The Quarterly Journal of Economics, 134(4), 2019, 2085-2134; NBER Working Paper 24343, https://doi.org/10.3386/w24343.

3. Ben Weidmann, Joseph Vecci, Farah Said, Sonia Bhalotra, Achyuta Adhvaryu, Anant Nyshadham, Jorge Tamayo, and David Deming, ‘How Do You Identify a Good Manager?’ The Quarterly Journal of Economics, 141(2), 2026, 1581-1633, https://doi.org/10.1093/qje/qjag004.

4. Edward P. Lazear, Kathryn L. Shaw, and Christopher T. Stanton, ‘The Value of Bosses,’ Journal of Labor Economics, 33(4), 2015, 823-861.

5. Todd J. Maurer and Manuel London, ‘From Individual Contributor to Leader: A Role Identity Shift Framework for Leader Development Within Innovative Organizations,’ Journal of Management, 44(4), 2018, 1426-1452, https://doi.org/10.1177/0149206315614372.

6. Allan Lee, Sara Willis, and Amy Wei Tian, ‘Empowering Leadership: A Meta-Analytic Examination of Incremental Contribution, Mediation, and Moderation,’ Journal of Organizational Behavior, 39(3), 2018, 306-325, https://doi.org/10.1002/job.2220.

7. Gavin R. Slemp, Margaret L. Kern, Kent J. Patrick, and Richard M. Ryan, ‘Leader Autonomy Support in the Workplace: A Meta-Analytic Review,’ Motivation and Emotion, 42, 2018, 706-724, https://doi.org/10.1007/s11031-018-9698-y.

8. Amy C. Edmondson, ‘Psychological Safety and Learning Behavior in Work Teams,’ Administrative Science Quarterly, 44(2), 1999, 350-383, https://doi.org/10.2307/2666999.

9. Kathleen Otto, Martin Baluku, Amelie Schaible, Cemre Oflu, and Emily Kleszewski, ‘The Only Way Is Up? How Different Facets of Employee and Supervisor Perfectionism Help or Hinder Career Development,’ Psychological Reports, published online 2024, https://doi.org/10.1177/00332941241229204.

10. Jim Harter, ‘Span of Control: What’s the Optimal Team Size for Managers?’ Gallup Workplace, January 14, 2026.

11. Randall Beck and Jim Harter, ‘Managers Account for 70% of Variance in Employee Engagement,’ Gallup Business Journal, April 21, 2015.

12. DDI, ‘New DDI Study Signals Looming Leadership Exodus, with 71% Reporting Increased Stress,’ Global Leadership Forecast 2025 press release, January 22, 2025.

13. Barbara B. Gaugler, Douglas B. Rosenthal, George C. Thornton III, and Cynthia Bentson, ‘Meta-Analysis of Assessment Center Validity,’ Journal of Applied Psychology, 72(3), 1987, 493-511, https://doi.org/10.1037/0021-9010.72.3.493.

Also, sometimes, it is the cheapest model in the room.

This sounds like the sort of contradiction vendors frame on a slide and finance teams circle in red. It is not actually a contradiction. It is a denominator problem. A model can cost more per token and less per solved task. Both statements can be true, and in production they often are.

The wrong debate is whether GPT-5.5 is cheap or expensive. The useful question is: expensive per what? Per input token? Per visible answer? Per successful workflow? Per avoided escalation? Per hour of engineering time not spent pursuing a result that was nearly correct in all the most inconvenient places?

That last denominator is where the tidy spreadsheet develops a cough.

At the cited standard API rates, the sticker shock is easy to understand: gpt-5.5 is priced at $5.00 per 1M input tokens, $0.50 per 1M cached input tokens, and $30.00 per 1M output tokens. GPT-5.4 is exactly half that on the same standard short-context tariff: $2.50, $0.25, and $15.00. GPT-4.1 is $2.00 for input, $0.50 for cached input, and $8.00 for output. GPT-4o is $2.50, $1.25, and $10.00.

So yes, if your entire model evaluation is a procurement table, GPT-5.5 looks expensive because it is expensive. The table has done its job and may now collect its certificate. But a production workflow also contains retries, tool mistakes, prompt scaffolding, hidden reasoning, caching, latency tiers, and human follow-up. Once those enter the calculation, the answer becomes much less obedient.

I think the simple rule is this: price per token tells you what the meter charges. Cost per success tells you whether the model belongs in the product.

* * *

The wrong denominator

A fresh request containing 10,000 uncached input tokens and 2,000 output tokens costs about $0.11 on GPT-5.5 at standard rates. The same token shape costs about $0.055 on GPT-5.4, $0.036 on GPT-4.1, and $0.045 on GPT-4o.

That is the cleanest argument against GPT-5.5. Hold the task shape fixed. Ignore quality. Ignore retries. Ignore whether the answer is usable. Under those rules, GPT-5.5 loses on direct request cost.

This is also how teams make bad AI architecture decisions using spreadsheets that look extremely responsible.

The real bill for a production feature is rarely one request. It is attempts per success. It is the extra prompt scaffolding added because the cheaper model needed handrails, road markings, and a responsible adult. It is the second call required after the first answer hallucinated a field name. It is the tool loop that searched the wrong source twice. It is the human reviewer who had to rescue the output. It is the user who quietly abandoned the workflow because the system was technically correct and operationally useless.

Price per token is not fake. It is merely incomplete, which is a more dangerous kind of correct.

If GPT-5.5 uses twice the tokens and succeeds at the same rate, it is more expensive. No financial origami required. If it uses half the total tokens because it needs less scaffolding, produces shorter useful answers, chooses tools correctly, and reduces retries, it can be cheaper despite the higher tariff. The arithmetic is dull. Measuring the right things is where the trouble begins.

This is where OpenAI’s own launch materials become economically interesting. OpenAI says GPT-5.5 improved on GPT-5.4 across Terminal-Bench 2.0, SWE-Bench Pro, and Expert-SWE while using fewer tokens across all three coding evaluations. In the system card, an external cyber evaluation by Irregular found that GPT-5.5 achieved higher success rates and lower cost per success than GPT-5.4, with CyScenarioBench cost per success falling by a factor of 2.7.

That is the useful evidence. Not “the model is smarter,” which is too vague to invoice. Not “frontier intelligence,” which is a fog machine wearing a conference badge. The useful claim is that, under at least one external task-level evaluation, the more expensive model was cheaper per successful completion.

Once you see that, the paradox loses most of its theatre. Expensive per token and cheap per completed task are not opposites. They are different measurements pretending to argue.

* * *

Hidden work is still work

The second trap is pretending that hidden reasoning is free because the user cannot see it.

Reasoning models use internal reasoning tokens before or between visible outputs. OpenAI’s reasoning documentation says these tokens are not exposed raw through the API, but they still occupy context-window space and are billed as output tokens. Depending on the task’s complexity, the model may use anywhere from a few hundred to tens of thousands of reasoning tokens.

This matters because a team can proudly remove verbose chain-of-thought from the visible answer while continuing to pay for a substantial amount of hidden computation. The invoice is admirably indifferent to whether a token was aesthetically pleasing.

The useful question is not “does GPT-5.5 reason?” It is “how much reasoning did this workflow purchase, and did the workflow need it?”

For a difficult codebase migration, deep research task, or multi-tool support agent, internal reasoning may be precisely what you are buying. For a routing classifier, it is probably a very elaborate toaster assigned to one slice of bread.

OpenAI’s GPT-5.5 migration guidance is essentially telling developers to stop preserving old prompt stacks as sacred architecture. The model defaults to medium reasoning effort. Low can be a better starting point for latency-sensitive workflows. High and xhigh should be earned through evaluations, not selected because they sound serious. The same guidance recommends setting text.verbosity to low when concise responses are sufficient and moving schema constraints into Structured Outputs rather than inflating the prompt.

This sounds obvious, but teams miss it all the time. They move to a stronger model, keep the old prompt shrine intact, leave reasoning effort too high, request long answers, and then complain that the model is expensive. It is expensive. They submitted a formal application for that outcome.

There is also a nastier edge case. If max_output_tokens is set too low, a reasoning model can spend tokens thinking and then return an incomplete response before producing meaningful visible output. OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when first experimenting with these models. That is not because every task needs 25,000 tokens. It is because starving a reasoning model can produce the worst possible bill: paid input, paid hidden work, and no useful answer.

The lesson is not “always lower reasoning.” The lesson is to make reasoning an explicit product setting. Pay for thought when thought changes the outcome. Do not pay for it because the dropdown looked important.

* * *

Caching is where the paradox gets boring

The most underrated economic lever is not a dazzling prompt technique. It is boring prefix hygiene.

Prompt caching happens when repeated prompt prefixes are processed once and reused. OpenAI says prompt caching can reduce latency by up to 80% and input-token costs by up to 90%, and that it works automatically for recent models including GPT-4o and newer. For GPT-5.5, extended prompt-cache retention is the default and can keep cached prefixes active for up to 24 hours.

This is why GPT-5.5 can look financially alarming in one workload and surprisingly reasonable in another.

Consider a context-heavy workflow with 100,000 cached input tokens, 1,000 live input tokens, and a 200-token answer. At standard prices, that shape costs about $0.061 on GPT-5.5. The same shape costs about $0.1295 on GPT-4o because GPT-4o’s cached-input rate is higher. GPT-4.1 remains slightly cheaper at about $0.0536, but the GPT-5.5 gap is no longer the horror story suggested by the fresh 10k/2k example.

Nothing mystical occurred. The denominator changed, and the token mix changed with it. Economics occasionally survives contact with arithmetic.

This is why I would not evaluate GPT-5.5 using a random chat benchmark and then declare a product strategy. If the real product has a large static prefix—policy documents, tool definitions, output contracts, examples, and customer-specific operating rules—then cache hit rate is part of the model decision.

A practical support-agent workflow might look like this.

Place stable instructions first: role, safety constraints, escalation rules, tool descriptions, schema, style contract, and the parts of the knowledge base reused across cases. Place dynamic content last: the current customer ticket, account state, recent messages, and the specific action requested. Use a consistent prompt_cache_key for prefixes that should share a cache. Measure cached_tokens on every call. Then test low and medium reasoning effort against the same representative tickets.

The evaluation should track first-pass resolution, number of tool calls, wrong-tool rate, retries per solved case, visible output tokens, reasoning tokens, cache hit rate, latency, and human escalation rate.

That is not a benchmark. That is a product instrument panel. The distinction matters because one helps you win a slide, while the other helps you operate the thing.

In this setup, GPT-5.5 does not need to be cheaper on every request. It needs to reduce total workflow cost while meeting the latency target. It can do that by using the cached prefix effectively, selecting tools more precisely, emitting less fluff, and avoiding the dead-end loops that make cheaper models mysteriously expensive.

* * *

There is still a workflow tax

The annoying part is that the model bill is not the whole bill. Apparently, software still requires engineering.

OpenAI’s GPT-5.5 guidance says to treat it as a new model family rather than a drop-in replacement for older GPT-5 prompts. That is not decorative migration prose. It means teams must retune prompts, reasoning effort, verbosity, tool descriptions, output contracts, and evaluations.

So GPT-5.5 can be cheaper per successful task and still cost more to adopt.

This is the part procurement tends to miss and engineering tends to describe as “a bit of tuning.” A model migration has labour cost. You need regression tests. You need representative examples. You need to compare answer quality rather than celebrating because the JSON parser did not burst into flames. You need guardrails for tool side effects. You need observability for reasoning tokens, cached tokens, and attempts per success. You need someone to decide when a task should be routed to GPT-5.5 rather than a cheaper model.

A stronger model does not eliminate architecture. It merely punishes lazy architecture more quietly.

The teams that achieve the best economics will not be the ones that put GPT-5.5 everywhere. They will route according to task value. Use cheaper models for low-risk classification, extraction, rewriting, and simple customer replies. Use GPT-5.5 where failure is expensive: ambiguous coding tasks, multi-step tool workflows, document-heavy analysis, research synthesis, incident triage, complicated customer-service cases, and jobs where a wrong first answer creates real work for someone else.

Preference is not performance. One team may prefer the cheaper model because its unit price is comforting. Another may prefer the frontier model because it feels magical. Neither preference has earned a seat in the calculation. What matters is the measured cost of a successful workflow.

The question should be painfully concrete: for this workflow, with this latency target, what is the cheapest model-and-orchestration combination that reaches the required success rate?

Sometimes the answer will be GPT-4.1. Sometimes GPT-4o. Sometimes a smaller model. Sometimes GPT-5.5 with low reasoning, low verbosity, aggressive caching, and Flex or Batch for non-urgent work. Sometimes GPT-5.5 Pro, although if you are using Pro everywhere without a very good reason, someone should gently take away your API key and make you write evaluations first.

* * *

A concrete way to evaluate it

Here is the workflow I would use before arguing about whether GPT-5.5 is expensive.

Choose one real production workflow, not a demo prompt groomed for applause. For example: “Given a customer ticket, inspect account state, search the policy base, decide whether to refund, draft the response, and route edge cases to a human.”

Build a small evaluation set from real cases. Keep it messy. Include ambiguous requests, missing information, policy conflicts, customers who use strange wording, and cases where the correct answer is to escalate. A neat evaluation set is how bad automation earns tenure.

Run the same cases through the candidate models. Keep the output contract identical. Do not let one model write a novel while another produces JSON and then call the comparison scientific. Track input tokens, cached input tokens, output tokens, reasoning tokens where available, tool calls, wrong tool calls, retries, latency, and whether an experienced operator would accept the final answer.

Then calculate cost in four ways.

First, cost per request. This is the number everyone reaches for because it is easy and behaves nicely in a cell.

Second, cost per accepted answer. This includes retries and failed attempts, which have an unfortunate habit of costing money despite being failures.

Third, cost per avoided human touch. This is where a model that looks expensive can become cheap very quickly.

Fourth, cost per product outcome under the latency target. If the model is cheap but misses the latency SLO, it is not cheap for that product. It is simply a slower method of disappointing users.

Only after those four numbers exist should anyone begin discussing model preference.

The useful pattern is usually hybrid. A cheap model handles simple routing. GPT-4.1 or GPT-4o handles straightforward structured work. GPT-5.5 handles cases where tool use, ambiguity, and persistence matter. Batch or Flex handles slow back-office enrichment. Prompt caching prevents the static context from becoming a fresh tax on every request. Verbosity stays low unless the user asks for an explanation. Reasoning effort rises only when evaluations prove that it buys success.

That architecture is less glamorous than placing the newest model behind every button. It is also less embarrassing when the invoice arrives.

* * *

The slightly annoying conclusion

GPT-5.5 is not cheap. That is the wrong comfort to seek.

It is a high-tariff model that can become economically rational when the task is hard enough, the prefix is cacheable enough, the output is concise enough, and the success-rate improvement is real enough. It is also a high-tariff model that can become a money fire when used for trivial tasks with reasoning effort left high, old prompt scaffolding preserved, and retries never measured.

The mistake is treating model cost as a property of the model alone. It is not. Model cost is a property of the model, the prompt, cache behaviour, orchestration, latency tier, failure rate, and the human workflow surrounding it.

That is why “GPT-5.5 is more expensive” and “GPT-5.5 is cheaper” can both be true. One person is counting tokens. The other is counting finished work.

The useful question is not whether the model is expensive.

The useful question is which denominator you are hiding behind.

* * *

Notes and References

1. OpenAI API Pricing. Current standard, Batch, Flex, Priority, tool-call, and regional-processing pricing. https://developers.openai.com/api/docs/pricing. Accessed 8 May 2026.

2. OpenAI API Compare Models. GPT-5.5 and GPT-5.4 pricing, context-window, max-output, endpoints, and model metadata. https://developers.openai.com/api/docs/models/compare. Accessed 8 May 2026.

3. OpenAI API Guide: Using GPT-5.5. Migration guidance, reasoning effort, verbosity, prompt structure, and benchmarking recommendations. https://developers.openai.com/api/docs/guides/latest-model. Accessed 8 May 2026.

4. OpenAI API Guide: Reasoning Models. Reasoning-token behaviour, billing, context-window management, and max_output_tokens guidance. https://developers.openai.com/api/docs/guides/reasoning. Accessed 8 May 2026.

5. OpenAI API Guide: Prompt Caching. Cache economics, static-prefix guidance, GPT-5.5 extended cache retention, and cached-token usage reporting. https://developers.openai.com/api/docs/guides/prompt-caching. Accessed 8 May 2026.

6. OpenAI API Guide: Flex Processing. Cost-latency trade-off and Batch-rate pricing for Flex requests. https://developers.openai.com/api/docs/guides/flex-processing. Accessed 8 May 2026.

7. OpenAI. Introducing GPT-5.5. Launch post, benchmark claims, token-efficiency statements, availability, and pricing overview. https://openai.com/index/introducing-gpt-5-5/. Accessed 8 May 2026.

8. OpenAI Deployment Safety Hub. GPT-5.5 System Card. Safety evaluation process, GPT-5.5 Pro test-time-compute note, and Irregular cost-per-success result. https://deploymentsafety.openai.com/gpt-5-5. Accessed 8 May 2026.

9. OpenAI API Model Page: GPT-4.1. Pricing, context window, and model characteristics used for comparison. https://developers.openai.com/api/docs/models/gpt-4.1. Accessed 8 May 2026.

10. OpenAI API Model Page: GPT-4o. Pricing, context window, and model characteristics used for comparison. https://developers.openai.com/api/docs/models/gpt-4o. Accessed 8 May 2026.

11. Artificial Analysis. GPT-5.5 (low) API provider benchmarking and price analysis. Latency, output speed, blended price, and methodology notes; figures are dynamic and should be rechecked before publication. https://artificialanalysis.ai/models/gpt-5-5-low/providers. Accessed 8 May 2026.

This sounds obvious, but teams miss it all the time. They debate which model is smarter, which coding agent has the better benchmark chart, and whether the paid tier gives them enough room. Meanwhile, the agent is quietly shovelling directory listings, dependency trees, test-runner confetti, repeated stack traces, polite throat-clearing, and five paragraphs of commentary into context where one useful line would have done.

The wrong debate is: how do I buy more room?

The useful question is: why am I filling the room with junk?

That is where tools like RTK and Caveman get interesting. Not because they are magic. Not because their headline compression numbers belong tattooed on your sprint board. They are interesting because they attack two different leaks in the agent loop.

RTK compresses terminal output before it becomes model input. Caveman compresses what the agent says back to you, and its companion compression workflow can reduce recurring instruction files. One sits at the command boundary. The other sits at the response and memory boundary.

Together, they suggest a better operating habit for AI-heavy development: treat tokens as working memory, not as disposable exhaust.

* * *

The token bill is not one thing

When people say “tokens,” they usually mean a vague unit of AI consumption. That vagueness is convenient. It is also how bad workflows hide.

OpenAI’s token documentation separates usage into input tokens, output tokens, cached tokens, and, for some advanced models, reasoning tokens. Tokens are not the same as words; the rough English rule of thumb is about four characters per token, with plenty of variation by language and text shape. [1]

For reasoning models, there is another trap: the thinking may not be visible, but it still matters. OpenAI’s reasoning-model documentation says reasoning tokens are not visible through the API, but they occupy context-window space and are billed as output tokens. It also notes that input and output tokens from each step carry over in a multi-step conversation, while reasoning tokens are discarded. [2]

Claude Code has the same practical warning in its cost guidance: thinking tokens are billed as output tokens, and the default budget can be tens of thousands of tokens per request depending on the model. Anthropic’s guidance says simpler tasks can use lower effort, disabled thinking, or a lower thinking budget. [3]

That gives us the first useful split.

There are tokens you send into the model. There are tokens the model generates back. There are hidden reasoning tokens. There are tokens retained across the conversation. Different tools affect different buckets.

Preference is not performance. A terse answer feels cheaper, but it does not necessarily reduce hidden reasoning. A compressed terminal log reduces context, but it can also hide the one line you needed. A smaller instruction file helps every session, but only if the compressed version preserves the rules that actually matter.

So the practical version is simple: RTK helps with CLI output before it pollutes context. Caveman helps with visible assistant output and repeated prose instructions. Neither should be described as a universal reasoning-token shrink ray.

That distinction matters because token hygiene is a systems problem, not a prompt-style problem.

* * *

RTK: cut terminal sludge before the agent reads it

AI coding agents love the terminal. They run tests. They inspect files. They check git status. They read logs. They list directories. They ask the shell for truth because the shell is where the project eventually stops being polite.

The problem is that most terminal output was designed for humans glancing at a screen, not for a language model carrying every character into context.

RTK, short for Rust Token Killer, is a CLI proxy that filters and compresses command output before it reaches the LLM context. Its README describes a single Rust binary with more than 100 supported commands and claims 60-90% token reduction on common development operations, with the caveat that actual savings vary by project size. [5]

The important part is not the exact percentage. The important part is the boundary.

Without a filter, the agent runs a command and receives raw output. With RTK, the agent runs the command through a proxy that can trim, group, truncate, and deduplicate before the response lands in the model’s working memory. The README describes four core strategies: smart filtering, grouping, truncation, and deduplication. [5]

That is boring in the best possible way.

A successful git push does not need the model to admire object enumeration, delta compression, thread counts, and progress bars. It needs to know whether the push worked, where it went, and whether anything unusual happened. RTK’s example turns a verbose git push into something like “ok main.” [5]

A failing test run does not need 200 lines proving that unrelated tests are having a lovely day. It needs the failing tests, their assertions, and enough location data to act. RTK’s test wrappers aim to show failures only for common runners such as pytest, cargo test, go test, Jest, Vitest, and Playwright. [5]

A directory listing does not need every permission bit unless permissions are the problem. A compact tree is often better than a full ls -la dump. A git status does not need the full decoration parade if the agent only needs modified, staged, and untracked files.

This is why RTK belongs early in the loop. It reduces the amount of machine chatter that becomes model input. It also makes later turns cleaner, because less irrelevant terminal text is retained in the conversation history.

There are sharp edges.

RTK’s auto-rewrite hook works by rewriting Bash commands to RTK equivalents. The README notes that Claude Code built-in tools such as Read, Grep, and Glob bypass that Bash hook, so those workflows are not automatically rewritten. On native Windows, RTK filters work but the hook does not auto-rewrite commands; WSL gets the fuller hook experience. [5]

That Windows note is not trivia. If your setup is PowerShell-first, you should assume explicit RTK commands are safer than relying on magic. Use rtk git status, rtk tsc, rtk pytest, rtk read, rtk grep. Make the compression visible enough that you know when it is active.

The discipline I would use is conservative: compress routine inspection, but keep a raw escape hatch. For exploratory file discovery, compressed output is usually fine. For build logs, failures-only is usually fine. For security review, incident response, flaky race conditions, or migration diffs, I want the full output available somewhere, even if the agent first sees the compressed view.

Token savings are useful. Losing the clue is expensive.

* * *

Caveman: make the agent stop narrating

Caveman attacks a different leak: the assistant’s habit of talking too much.

The Julius Brussee Caveman project is a skill/plugin pattern for AI coding agents that pushes replies into terse, compressed language. Its README describes triggers such as /caveman, “talk like caveman,” “caveman mode,” and “less tokens please.” It also includes intensity levels: lite, full, ultra, and Wenyan modes. [7]

The README reports benchmark results across common coding tasks, with an average reduction from 1,214 normal output tokens to 294 Caveman tokens, or about 76% saved, across the listed prompts. The reported range is 22% to 87%. [7]

Treat those numbers as project-reported measurements, not universal law. The more useful claim is operational: less visible assistant prose means less output to pay for, less text to read, and less previous-assistant chatter carried into later context.

This is not just about being cute. Agent replies are often padded with softeners: “Here’s what I found,” “The issue appears to be,” “Let me know if you’d like,” “We should probably consider,” and so on. In a normal chat, that can make the interaction feel nicer. In an engineering loop, it is often ceremony wearing a token meter.

A good agent response during coding is usually closer to this:

auth.ts: token expiry bug. Cause: refresh path skips clock skew guard. Fix: add 60s leeway before reject. Tests: auth-refresh.spec.ts failing before, passing after.

That is not caveman because it is silly. It is caveman because it is a compact status packet.

The crucial correction: Caveman does not shrink hidden thinking tokens. The README says this directly: Caveman only affects output tokens; thinking and reasoning tokens are untouched. [7]

That matters because the phrase “compress thinking and output” is tempting but inaccurate. Caveman can make the agent’s visible answer shorter. It can make recurring memory or instruction files shorter through caveman-compress. It can reduce later context because previous visible assistant replies are shorter. But if the model is doing extended reasoning internally, that budget is controlled elsewhere.

In Claude Code, that “elsewhere” is things like effort level, model settings, thinking configuration, and thinking-token budget. Anthropic’s cost guidance is explicit about adjusting extended thinking for simpler tasks. [3]

Caveman also has an input-side trick worth separating from the response style. Its caveman-compress workflow rewrites memory files such as CLAUDE.md into compressed prose while keeping a human-readable original backup. The README’s sample table reports an average reduction from 898 to 481 tokens, or 46%, across example files. It says code blocks, URLs, file paths, commands, headings, dates, and version numbers pass through untouched while prose is compressed. [7]

That is more interesting than the meme suggests.

Claude Code documentation says CLAUDE.md loads into context at session start. If that file contains detailed instructions for workflows you are not using, those tokens are present anyway. Anthropic recommends moving specialised instructions into skills because skills load on demand, and aiming to keep CLAUDE.md under 200 lines. [3]

So the stronger pattern is not “make every answer caveman forever.” The stronger pattern is: keep base context lean, load specialised instructions only when needed, and make routine status updates compact by default.

There is a trade-off. Over-compression can hide uncertainty. It can also make an agent sound more confident than it should, because hedging gets deleted along with filler. That means any caveman-style rule should preserve three things: blockers, confidence, and next action.

Cut ceremony. Keep doubt.

* * *

A practical token-stretching workflow

Here is the workflow I would actually try for an AI coding setup. Nothing heroic. No dashboard. No spreadsheet theatre. Just fewer stupid tokens.

First, baseline the obvious leak. Run a normal 30-minute agent session and note where the agent reads too much. Not the total cost. The shape. Did it repeatedly call tree? Did it dump long test logs? Did it read whole files to find one function? Did it explain every tiny step in paragraphs?

Second, put RTK on the shell path and use it explicitly for high-noise commands. Start with status, diff, test, lint, grep, file read, logs, and dependency inspection. On WSL or Unix-like shells, experiment with the auto-rewrite hook. On native Windows, assume explicit commands until you have verified the hook path you actually use.

Third, give the agent a rule for escalation. Something like: use compressed output for routine inspection; request raw output when a failure is ambiguous, security-sensitive, or caused by timing, encoding, permissions, or environment state. The agent should know compression is a first pass, not the court transcript.

Fourth, turn on Caveman-style terse responses for routine engineering loops. Do not ask for a personality change. Ask for a reporting format. File. Cause. Fix. Test. Risk. Next step. That is enough for most turns.

Fifth, compress recurring instruction files only after you decide what belongs there. This is where people get it backwards. They take a bloated CLAUDE.md, compress it, and declare victory. Better: delete stale rules, move workflow-specific guidance into skills, then compress the small durable core.

Sixth, tune reasoning separately. For planning, architecture, migrations, and unfamiliar bugs, keep higher reasoning. For “rename this field,” “fix lint,” “update tests,” or “summarise this diff,” lower effort may be enough. This is not a moral issue. It is matching compute to task shape.

A concrete example:

The agent needs to fix a TypeScript auth bug. In the noisy version, it runs a full directory tree, cats several files, runs the full test suite, receives pages of test output, explains its plan in six paragraphs, edits one file, runs the full test suite again, then writes a novella about what changed.

In the cleaner version, it starts with rtk grep around the token expiry path, uses rtk read on the relevant files, runs the focused auth test through an RTK test wrapper, and reports in a terse packet: auth expiry bug. Cause: refresh branch ignores leeway. Changed auth/session.ts. Focused test passes. Full suite not run.

That last sentence is important. Compression should not mutate reality. If the full suite was not run, the agent must say so. Brevity is useful only when it preserves the edges.

For longer tasks, delegate noisy operations. Claude Code’s cost guidance says running tests, fetching docs, or processing logs can consume significant context, and recommends delegating verbose operations to subagents so only a summary returns to the main conversation. [3]

That pattern composes with RTK and Caveman. RTK trims the subagent’s tool output. The subagent returns a compact summary. Caveman keeps the main agent from turning that summary into a speech.

The shape is: raw world, compressed tool signal, compact agent status, explicit raw escape hatch.

That is the whole game.

* * *

Free tools still need adult supervision

The obvious failure mode is believing the compression number more than the work.

A tool can save 80% of tokens and still be the wrong tool for the moment. A directory listing compressed into a tree is helpful. A stack trace compressed until the causal frame disappears is not. A terse PR review can be excellent. A terse security review can become malpractice if it drops evidence.

So I would classify compression targets by risk.

Low risk: successful command confirmations, dependency list summaries, passing test noise, repeated log lines, boilerplate install output, directory discovery, generic status updates.

Medium risk: failing tests, compiler errors, diffs, migration logs, build errors, dependency conflicts. Compress first, but preserve enough line numbers, filenames, error codes, and command context to act.

High risk: security findings, data-loss operations, production incidents, permissions, auth, cryptography, billing, legal text, compliance evidence, and any bug where one missing line changes the diagnosis. For these, compression should be an index, not the evidence.

The same applies to Caveman. I like terse agents. I do not like agents that delete caveats. The goal is not to sound primitive. The goal is to remove prose that does not carry state.

A good compressed engineering reply still includes uncertainty markers. Likely cause is different from cause. Need full log is different from fixed. Focused test passes is different from all tests pass. These are not filler words. They are control signals.

This is where the meme can accidentally teach the wrong lesson. The point is not to make the model dumber. The point is to make the channel cleaner.

There is some research support for brevity as more than aesthetics, but it should be handled carefully. A March 2026 arXiv preprint found that, on certain benchmark subsets, constraining large models to brief responses improved accuracy and reduced overelaboration errors. [8]

That does not mean every task wants fewer words. It means verbosity is not automatically intelligence. In engineering work, verbose can be worse than terse when it smears the state of the system across polite paragraphs.

The rule I would use: compress transport, not truth.

* * *

The better question

The useful question is not “Which AI plan gives me more tokens?”

It is “Where are my tokens leaking?”

Some leak through terminal output. Use RTK or the same idea: filter command output before the model sees it.

Some leak through assistant verbosity. Use Caveman or the same idea: make routine responses state-dense.

Some leak through oversized base instructions. Move specialised rules into on-demand skills. Keep the default context small.

Some leak through unnecessary reasoning. Lower effort when the task is simple. Raise it when the task deserves it.

Some leak through bad task framing. Ask for the file, function, test, or failure mode you care about instead of telling the agent to “look around.” “Look around” is how an agent ends up inventorying the furniture.

None of this requires a new religion. It requires a small amount of mechanical sympathy for how agent loops actually burn context.

The future of AI coding will probably include larger windows, better caching, smarter tool calling, and more automatic compression. Anthropic has already shipped token-efficient tool-use features for the API that reduce output token consumption for tool calls, with early users seeing an average reduction of 14% and up to 70% in some cases. [9]

But even if the tools get smarter, the habit still matters.

A bigger context window is useful. A cleaner context window is leverage.

So yes: free tools to help you stretch your tokens. RTK for terminal noise. Caveman for agent chatter and recurring prose. Keep the claim honest: RTK compresses CLI output before it becomes model input; Caveman compresses visible output and can compress memory prose, but it does not directly compress hidden thinking tokens.

The slightly annoying question is the one worth keeping on your desk:

Are you running out of tokens, or are you just giving the model too much trash to carry?

* * *

Notes and References

1. OpenAI Help Center, “What are tokens and how to count them?”, accessed April 27, 2026. https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them

2. OpenAI API documentation, “Reasoning models”, accessed April 27, 2026. https://developers.openai.com/api/docs/guides/reasoning

3. Anthropic Claude Code documentation, “Manage costs effectively”, accessed April 27, 2026. https://code.claude.com/docs/en/costs

4. Anthropic Claude API documentation, “Token counting”, accessed April 27, 2026. https://platform.claude.com/docs/en/build-with-claude/token-counting

5. rtk-ai/rtk GitHub README, accessed April 27, 2026. https://github.com/rtk-ai/rtk

6. RTK website, “Make your AI coding agent smarter”, accessed April 27, 2026. https://www.rtk-ai.app/

7. JuliusBrussee/caveman GitHub README, accessed April 27, 2026. https://github.com/JuliusBrussee/caveman

8. MD Azizul Hakim, “Brevity Constraints Reverse Performance Hierarchies in Language Models”, arXiv:2604.00025, submitted March 11, 2026. https://arxiv.org/abs/2604.00025

9. Anthropic, “Token-saving updates on the Anthropic API”, March 13, 2025, accessed April 27, 2026. https://claude.com/blog/token-saving-updates

10. A verification note: the article treats RTK and Caveman savings figures as project-reported measurements unless stated otherwise. I could verify the claims as published by the projects, but not independently reproduce their benchmark results during this pass.

How many people can this replace? How many hours can it shave? How much output can we squeeze from the same org chart before the org chart starts making a noise?

I think that is the wrong debate.

The useful question is not whether AI can make average work cheaper. It can. The useful question is whether it makes serious attempts cheap enough that teams can afford to search better.

That sounds like a small distinction.

It is not.

Treat AI as cheap labor and you get a faster content mill, a larger backlog of plausible nonsense, and a shiny new category of review debt. Treat AI as cheap ideation and critique and you can raise the quality frontier. You get more candidate designs, more test cases, more product angles, more copy variants, more user stories, more failure modes, and more chances to find the one option worth shipping.

The problem is usually not generation.

It is selection.

Cheap Means Marginal, Not Magical

When people say AI is cheap, the word cheap often acquires several additional jobs.

Cheap to deploy. Cheap to govern. Cheap to integrate. Cheap to trust.

That larger claim is false.

AI is cheap in a narrower and more interesting sense: it lowers the marginal cost of one more attempt.

Stanford HAI’s 2025 AI Index reported that the inference cost for a system performing at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024. That is the part of the economics that matters for ideation. One more draft, summary, test case, sketch, critique, or scenario is suddenly much cheaper than it used to be. [1]

But the model call is not the program cost.

The total cost still includes workflow redesign, data access, retrieval, security review, evaluation harnesses, legal controls, vendor management, and the human time required to decide whether the output is any good. The productivity J-curve literature makes the same point in language less likely to appear on a keynote slide: general-purpose technologies usually need complementary intangible investments before their benefits show up cleanly. [2]

This is why some AI pilots feel miraculous and others feel like expensive theater with excellent autocomplete.

The difference is not only model quality. It is whether the organization has built the surrounding machinery that converts cheap attempts into better outcomes.

Cheap attempts without selection are noise.

Cheap attempts with disciplined selection are leverage.

The Thing AI Buys Is Search

A lot of teams still ask the small question:

Can AI write this memo?

The better question is:

How many versions of the memo, using how many frames and representing how many stakeholder perspectives, can I now afford to inspect before making the decision?

That is the strategic shift. AI changes the search budget.

In a professional writing experiment published in Science, ChatGPT reduced task time by 40% and increased output quality by 18%. That is not merely a speed story. It is also a quality story, because the tool made drafting and revision cheaper inside a bounded task. [3]

The Boston Consulting Group field experiment is even more useful because it reveals both the upside and the trap. In the experiment, 758 consultants completed realistic consulting tasks. For tasks inside the model’s capability frontier, consultants using AI completed 12.2% more tasks, worked 25.1% faster, and produced work rated more than 40% higher in quality than the control group. For one task outside the frontier, however, consultants using AI were 19 percentage points less likely to produce the correct answer. [4]

That is the entire AI strategy problem in miniature.

Inside the frontier, AI buys more search.

Outside the frontier, AI buys false confidence at scale, now available with professional formatting.

The same pattern appears elsewhere. In customer support, a study of 5,179 agents found that access to a generative AI assistant increased productivity by 14% on average, with a 34% improvement among novice and low-skilled workers. That matters because AI did not simply replace expertise. It helped redistribute some of the practices of stronger workers to people with less experience. [5]

In product innovation work at Procter & Gamble, a field experiment involving 776 professionals found that individuals using AI matched the performance of two-person teams working without AI. It also found that AI helped R&D and commercial professionals produce more balanced solutions across functional boundaries. [6]

That is not primarily a headcount story.

It is a collaboration story.

The strongest version of this argument is not that AI makes experts unnecessary. It is that AI makes a broader pool of ideas available before experts spend their scarce time.

Creative work shows the same promise and the same bill arriving afterward. In a Science Advances study, writers with access to AI-generated ideas produced stories judged more creative, better written, and more enjoyable. But the collective diversity of those stories fell. [7]

That final sentence is the tax.

AI can raise the floor while quietly moving the walls inward.

In R&D, the search argument becomes more literal. Google DeepMind’s GNoME work, published in Nature, reported 381,000 new stable crystal structures available on the convex hull. The point is not that a scientist typed faster. The point is that AI changed the scale at which candidates could be explored. [8]

This is why “AI as automation” is too small a frame.

For many teams, the practical advantage is not doing the same thing with fewer people. It is making it economically reasonable to try more things, compare them, and find a better one.

The Jagged Frontier Is Where ROI Gets Embarrassed

The lazy pro-AI story says the tool makes everyone faster.

The lazy anti-AI story says the tool is mostly slop.

Both are wonderfully convenient.

What actually matters is task fit. The frontier is jagged. A model may be excellent at drafting a first-pass policy, mediocre at designing the policy’s exception system, and dangerous when asked to infer the legal status of an edge case without grounding.

This sounds obvious. Teams miss it anyway.

They run one impressive demo, generalize from it, and then act surprised when a different workflow folds itself neatly into a production incident.

Software is a good place to practise humility. AI coding tools can be valuable for bounded tasks, scaffolding, test generation, and unfamiliar syntax. They can also slow down experts working inside large, mature codebases where context, judgment, and verification dominate the work.

METR’s early-2025 randomized trial found that experienced open-source developers working in their own repositories took 19% longer when AI tools were allowed. METR later noted that a follow-up experiment in late 2025 and early 2026 became harder to interpret because many developers did not want to participate in work without AI and because task selection changed. The researchers said the later data provided weak evidence of speedup rather than a clean estimate. [10]

The lesson is not “AI slows developers.”

The lesson is that measurement without context is a prank wearing a chart.

Advertising evidence has the same jagged shape. In the Pairit field experiment, human-AI teams produced 50% more ads per worker and higher-quality text, while human-human teams produced higher-quality images. The AI-assisted outputs also became more homogeneous. [9]

So yes, AI can produce more.

Sometimes it produces better.

Sometimes it produces faster sameness.

Preference is not performance.

A user liking the tool is not the same as the work improving. A team feeling faster is not the same as the release becoming safer. A stack of polished drafts is not the same as a better decision, although it does occupy more tabs.

AI often moves cost from generation to review.

That is fine, provided you admit it.

The question is not, “Did we produce more?”

The question is, “Did the extra attempts improve the selected output after evaluation, risk controls, and downstream consequences?”

That is a much harder question.

It is also the only one worth asking.

A Workflow: Generate Broadly, Critique Narrowly, Approve Explicitly

Here is the pattern I would use for most quality-led AI pilots.

Do not begin by buying the fanciest model for everyone and hoping organizational wisdom appears in the license portal.

Begin with one workflow where additional variants could plausibly improve the outcome and where quality can be measured.

Product onboarding. Campaign copy. Support macros. Test generation. Architecture decision records. Internal research briefs. Design concepts. Customer email drafts.

Then run the workflow like this.

1. Define the quality metric before anyone touches the model

For a marketing team, that might include click-through rate, conversion rate, cost per acquisition, brand compliance, and creative diversity.

For software, it might include unit-test pass rate, review churn, revert rate, defect escape, maintainability, cycle time, and developer satisfaction.

For product design, it might include feasibility, novelty, variety across solution families, user value, and downstream validation success.

Developer productivity frameworks such as SPACE explicitly warn against reducing productivity to a single activity metric. Design research similarly distinguishes quantity, quality, novelty, and variety in ideation. [11] [12]

A metric chosen after the demo is not really a metric.

It is applause with arithmetic.

2. Generate too many options on purpose

Ask for 30 candidate onboarding flows, not three.

Ask for failure modes affecting novice users, power users, support teams, legal reviewers, and abuse cases. Ask for boring ideas, risky ideas, high-variance ideas, and ideas optimized for constraints the team normally ignores.

This is where cheap models, templates, and reusable prompts make sense.

You are not looking for the final answer.

You are buying breadth.

3. Force disagreement

Have the model critique its own outputs against the rubric. Have a second model argue against the first. Have humans mark outputs as promising, wrong, duplicative, risky, or surprisingly useful.

Ask what assumptions would need to be true for each option to win.

This is not because models are reliable judges.

They are not.

It is because structured critique turns a pile of drafts into a map of trade-offs. The pile is still there, but at least it now has road signs.

4. Ground the shortlist

This is where teams should spend money.

Use retrieval against approved sources. Run tests. Check facts. Ask legal or security to review only the shortlisted options that genuinely matter. Use stronger models for difficult reasoning, not for every passing brainstorm that wandered into the prompt.

Put humans at the points where judgment, accountability, and taste matter.

5. Approve explicitly

The final artifact should include a short decision log:

• What was selected

• Why it was selected

• What was rejected

• What evidence was used

• What risks remain

• Who owns the outcome

That log is not bureaucracy.

It is institutional memory, which is what bureaucracy occasionally becomes when it is having a good day.

Consider a concrete example. Suppose a product team wants to improve trial activation.

The weak AI workflow asks a chatbot for onboarding ideas and ships the prettiest one.

The useful workflow asks for 40 onboarding variants across user segments, clusters them into themes, and scores them against friction, clarity, implementation cost, abuse risk, and likely activation impact. The team then selects three for testing.

Engineering uses AI to generate edge cases and test plans. Support uses AI to draft changes to its macros. Legal reviews only the two flows that touch sensitive data. Product records why the winner was chosen.

No one in that workflow worships the model.

They use it to make exploration cheap, then spend human attention where it compounds.

Measurement Is the Moat

AI programs fail quietly when they measure activity and call it value.

Number of prompts is not value.

Number of drafts is not value.

Tokens consumed is not value.

Hours allegedly saved are not value unless those hours become something better.

The target is quality-adjusted output.

That phrase is ugly.

It is also the right one.

A quality-led ROI model should include four terms:

• What was the value of the quality uplift?

• What additional experiments became possible?

• What time savings were actually redeployed to higher-value work?

• What risk losses were avoided?

Then subtract the total AI program cost, not merely the model invoice, which is usually the friendliest number in the room.

The total cost includes seats, tokens, integration, retrieval, observability, evaluations, security, compliance, human review, training, vendor management, and the time spent cleaning up confident mistakes.

The failure mode is predictable. A team runs a pilot, sees that people feel faster, buys more licenses, and declares victory before measuring whether the selected outputs improved.

This is why the most useful AI metrics are often boring.

For software, count fewer escaped defects, fewer reverts, faster review of low-risk changes, better test coverage, and less review fatigue.

For marketing, count more meaningful experiments and better conversion, not merely more copy.

For product, count whether the team found ideas that survived contact with user evidence.

For internal writing, count whether decisions became clearer and faster, not whether the documents became longer and developed an executive summary.

The best AI deployment will often look less like a productivity dashboard and more like an experiment system.

Baseline.

Generate.

Select.

Test.

Learn.

Repeat.

Governance Is Not a Vibes Tax

A lot of builders treat governance as the department of no.

That is lazy.

Risk-based governance is how you stop cheap attempts from becoming expensive mistakes.

NIST’s Generative AI Profile for the AI Risk Management Framework identifies risks directly relevant to this kind of workflow: data privacy, harmful bias and homogenization, intellectual property, information integrity, and the need for pre-deployment and ongoing evaluation. [13]

Those are not abstract enterprise concerns floating somewhere above the real work.

They appear inside ordinary work.

A model drafts customer copy that subtly changes a claim. A sales deck includes an invented customer reference. A code assistant suggests a dependency with a licensing problem. A team uses the same model and prompt pattern until every campaign sounds as though it came from the same mildly caffeinated intern. A product researcher uploads sensitive notes into a tool that should never have seen them.

Cheap ideation increases surface area.

Governance should therefore live inside the workflow, not arrive at the end carrying a clipboard and bad news.

That means maintaining a use-case inventory. Setting clear data boundaries. Using retrieval from approved sources. Defining human-review thresholds. Recording provenance for generated assets. Keeping evaluation logs. Supporting incident reporting. Reviewing vendors. Periodically checking for output sameness.

It also means having a rule that high-stakes uses require stronger evidence than “the model sounded confident.”

Copyright belongs in this discussion too. The U.S. Copyright Office’s 2025 report on AI and copyrightability concluded that purely AI-generated material is not copyrightable, that prompts alone are unlikely to provide sufficient control, and that human-authored expression, creative selection, arrangement, or modification can matter. [14]

The practical point is simple:

If the output matters, document the human contribution.

Not because paperwork is fun.

Because provenance becomes part of quality.

The Uncomfortable Team Question

If AI makes attempts cheap, the role of the team changes.

People do less blank-page production. They do more framing, delegation, evaluation, verification, and integration.

That sounds tidy.

In practice, it is politically weird.

The person once rewarded for producing the first draft may now be rewarded for writing the best rubric. The senior engineer may spend less time typing boilerplate and more time identifying which generated patch is subtly wrong. The product manager may become more valuable by asking better questions than by producing longer strategy documents. The designer may become the guardian of taste against the beige flood.

This is why AI adoption is an organization-design problem disguised as a tool rollout.

DORA’s 2025 research on AI-assisted software development put the point cleanly: AI acts as an amplifier, magnifying an organization’s existing strengths and weaknesses. The greatest returns come not from the tools alone, but from improving the underlying system. [15]

That matches the evidence everywhere else.

AI helps when the workflow knows what good means.

It hurts when the workflow has no taste, no metrics, no review capacity, and nobody willing to say no to polished garbage.

The wrong debate is whether AI replaces people.

The sharper debate is which human work becomes more important when generation becomes cheap.

My answer is framing, taste, domain judgment, experimental discipline, and accountability.

AI is not cheap intelligence.

It is cheap attempts.

The teams that benefit will not be those producing the largest volume of synthetic output. They will be the teams that use cheap attempts to find better options, then retain the discipline to reject most of them.

So here is the slightly annoying question.

Where is your team still behaving as though serious attempts are expensive?

And is that prudence, or merely an old cost structure wearing a strategy badge?

Notes and References

1. Stanford Institute for Human-Centered Artificial Intelligence, The 2025 AI Index Report, 2025. https://hai.stanford.edu/ai-index/2025-ai-index-report

2. Erik Brynjolfsson, Daniel Rock, and Chad Syverson, “The Productivity J-Curve: How Intangibles Complement General Purpose Technologies,” American Economic Journal: Macroeconomics, 2021. https://www.aeaweb.org/articles?id=10.1257/mac.20180386

3. Shakked Noy and Whitney Zhang, “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence,” Science, 2023. https://www.science.org/doi/10.1126/science.adh2586

4. Fabrizio Dell’Acqua et al., “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality,” Harvard Business School / SSRN, 2023. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

5. Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, “Generative AI at Work,” NBER Working Paper 31161, 2023; later published in The Quarterly Journal of Economics, 2025. https://www.nber.org/papers/w31161

6. Fabrizio Dell’Acqua et al., “The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise,” NBER Working Paper 33641, 2025. https://www.nber.org/papers/w33641

7. Anil R. Doshi and Oliver P. Hauser, “Generative AI Enhances Individual Creativity but Reduces the Collective Diversity of Novel Content,” Science Advances, 2024. https://www.science.org/doi/10.1126/sciadv.adn5290

8. Amil Merchant et al., “Scaling Deep Learning for Materials Discovery,” Nature, 2023. https://www.nature.com/articles/s41586-023-06735-9

9. Hyelim Ju et al., “Collaborating with AI Agents: A Field Experiment on Teamwork, Productivity, and Performance,” arXiv, version updated 2026. https://arxiv.org/abs/2503.18238

10. Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” METR, 2025; and Joel Becker et al., “We are Changing our Developer Productivity Experiment Design,” METR, 2026. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ and https://metr.org/blog/2026-02-24-uplift-update/

11. Nicole Forsgren et al., “The SPACE of Developer Productivity,” ACM Queue, 2021. https://queue.acm.org/detail.cfm?id=3454124

12. Scarlett R. Miller et al., “How Should We Measure Creativity in Engineering Design? A Comparison of Social Science and Engineering Approaches,” Journal of Mechanical Design, 2021. https://asmedigitalcollection.asme.org/mechanicaldesign/article/143/3/031404/1090546/How-Should-We-Measure-Creativity-in-Engineering

13. National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

14. U.S. Copyright Office, Copyright and Artificial Intelligence, Part 2: Copyrightability, 2025. https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-2-Copyrightability-Report.pdf

15. DORA, State of AI-assisted Software Development, 2025. https://dora.dev/dora-report-2025/

After a long day, you stop failing heroically and start failing stupidly. You lose the thread of a conversation that is still happening. You reread the same paragraph. You forget the constraint you wrote down an hour ago, presumably to prevent exactly this.

Then you watch a long-running agent do the machine version of the same thing. It misses the key detail buried in the middle of the prompt. It keeps circling a dead end. It summarizes a tool result so aggressively that the only useful fact disappears. It is not malicious. It is overloaded.

People want to turn this into philosophy. Are humans just next-token predictors? Are LLMs baby minds? I think that is the wrong debate.

The useful question is not whether humans and models are the same kind of thing. It is whether they share enough operational constraints that the comparison makes us better builders. On that question, the answer is yes.

The big similarity is boring and extremely useful: both humans and LLM systems have a limited active workspace sitting in front of a much larger store of potential information. When that workspace gets cluttered, performance gets weird.

That is why the analogies land. Brain fog and context rot feel related. Note-taking and RAG feel related. Bad evidence poisoning human recall and bad retrieval poisoning agent output feel related.

But only if you keep the guardrails on. Human fatigue is biological. Human memory changes during sleep. LLM memory is usually an engineered stack of context, summaries, retrieval, logs, and perhaps a persistent memory layer. Treat the analogy as workflow, not metaphysics, and it becomes surprisingly sharp.

* * *

The wrong debate is ontology

Human working memory is small. Depending on the frame, researchers put the core limit around roughly four chunks, or about three to five meaningful chunks. That is not a lot of live slots for planning, holding constraints, and comparing alternatives.

LLMs have the same kind of bottleneck in a different substrate. The context window is the obvious version, but the deeper issue is not merely how much text fits. It is how much of that text the model can still use well.

This sounds obvious, but teams miss it all the time. They hear “128K” or “1M tokens” and mentally translate that into “the model can keep everything in mind.” No. A larger whiteboard is not the same thing as better working memory. Sometimes it is just a larger whiteboard with more things written in the wrong corner.

The Lost in the Middle paper made this painfully concrete: models often perform best when relevant information sits near the beginning or end of a long prompt, and worse when the key fact is buried in the middle. NoLiMa pushed the point further by stripping away easy literal matches. Performance fell sharply as contexts grew, even before models hit their advertised limits.

So the useful analogy is not “humans are language models.” It is “both systems have a fragile active state.” What actually matters is not raw storage. It is selection, ordering, salience, and refresh.

That is also why context engineering is not prompt decoration. It is workload design. You are deciding what gets desk space right now, what gets pushed into a filing cabinet, what gets summarized, and what gets dropped.

Preference is not performance. Developers prefer the fantasy that one giant context will remove the need for memory architecture. Performance usually comes from the opposite move: smaller working sets, tighter task framing, and more aggressive eviction of stale state. Apparently “keep the desk tidy” survives the transition to artificial intelligence.

* * *

Brain fog is context debt

When people say brain fog after a long day, they are usually pointing at slower thinking, weaker attention, and worse executive control after sustained cognitive effort. Reviews of cognitive fatigue describe exactly that sort of degradation.

The model-side analogue is real, but it is not literal fatigue. Models do not need coffee. They accumulate context debt.

Context debt appears when a session becomes an archaeological site: too many tool results, too many half-resolved threads, too many summaries of summaries, too much retrieved material with no ranking discipline. The agent can still “see” a lot of text. It just stops allocating attention to the right parts of it.

This is why giving the model a break does nothing unless the break changes the state. A human benefits from sleep, food, a walk, or simply stopping. An agent benefits from pruning, compaction, retrieval, and explicit memory writes. Sending it away for the weekend would mostly postpone the next API call.

Anthropic’s own engineering work makes the point in a very unromantic way. In a 2025 internal agentic-search evaluation, context editing alone improved performance by 29 percent over baseline, and context editing plus a memory tool improved it by 39 percent. In a 100-turn search evaluation, context editing also cut token consumption by 84 percent while allowing workflows to complete that would otherwise fail from context exhaustion.

That is the punchline. The lever is not empathy for the model. The lever is state management.

I think this matters because a lot of teams still debug agents as if the main problem were intelligence. Sometimes it is. Much more often the agent is doing the exact wrong thing with the exact wrong pile of context that you handed it.

* * *

Memory is compression, not recording

Human memory is not a replay buffer. It is reconstructive. Reviews of hippocampal-neocortical memory transformation describe a shift from detail-rich traces toward gist-like and schematic representations over time. Sleep-dependent replay is thought to help the hippocampus teach the neocortex, turning specific episodes into more reusable knowledge.

That is a terrible metaphor for most production AI memory features, which is exactly why people keep getting confused.

In most deployed systems, “memory” means one of four things: model weights, current context, a retrieved external store, or a persistent note that can be reinserted later. The original RAG paper called this split parametric memory versus non-parametric memory. That distinction still does a lot of useful work.

OpenAI’s current memory documentation is explicit that saved memories are stored separately from chat history. Anthropic’s current memory tooling is explicit in the other direction: the memory tool is client-side, and developers control where the data is stored. In both cases, the practical picture is much closer to a notebook plus a recall policy than to anything like autonomous consolidation.

So when teams say, “the model learned this during the conversation,” I usually flinch. Unless you retrained or fine-tuned something, it probably did not learn in the human sense. It wrote a better sticky note.

This is also where note-taking becomes a stronger analogy than people realize. The classic note-taking literature distinguishes between an encoding effect and an external-storage effect. Writing notes can help you learn. Reviewing notes later also helps you remember.

Agent memory writes usually get only half of that. They are storage plus future lookup. Human note-taking is storage plus self-training.

That asymmetry explains why dumping every observation into a vector store feels disappointing. You built retention, not understanding. The filing cabinet is magnificent. Nobody became wiser.

The useful question is not whether to use long context or RAG as a religion. Recent comparisons suggest that long context can outperform RAG when you spend enough tokens, especially on question answering over dense sources, while RAG keeps real cost advantages and can still win in some dialogue or general-query settings. Anthropic’s Contextual Retrieval work also showed how much retrieval quality matters: its method reduced failed retrievals by 49 percent, and by 67 percent when combined with reranking.

What actually matters is tiering. Stable policy belongs in persistent memory. Source material belongs in retrieval. Immediate objectives belong in the live working set. Decisions and unresolved issues belong in a rolling summary. And facts you want the base model to truly internalize belong in training, not in wishful thinking.

* * *

Bad evidence scales faster than good memory

There is one more similarity that matters because it wrecks outputs quietly: both humans and models are vulnerable to bad evidence.

The misinformation effect literature showed long ago that people can absorb false post-event information about something they actually experienced and later recall the corrupted version. That is one of the clearest reminders that human memory is reconstructive, not camera-like.

Agent systems have the machine version of the same problem. A retrieved chunk can be irrelevant, noisy, outdated, or simply false. Once it enters the working set, the model often treats it as the problem definition. The right answer can still exist somewhere in the system, and the run can still fail because the active evidence stream got polluted.

This is why provenance is not paperwork. It is part of reasoning quality.

Anthropic ran into the practical version of this while building its research system: human testers found that early agents sometimes preferred SEO-optimized content farms over better sources. That is not a cute bug. That is the whole ballgame. If the retrieval layer rewards the wrong evidence, the model will faithfully produce polished nonsense from the wrong pile of facts.

A lot of teams respond by adding more memory. Usually that makes it worse. Bad evidence that gets summarized, stored, and reintroduced later stops being a transient mistake and becomes durable contamination. Congratulations: the bug now has institutional memory.

The useful rule is simple. Never let unaudited retrieval jump straight into persistent memory. First rank it. Then attribute it. Then compare it against an independent source if the claim matters. Only then let it harden into summary state or a saved fact.

For humans, the equivalent is obvious. We do better when we write down where a claim came from, compare notes, and correct errors quickly before they become the story we now remember. For agents, the same discipline applies, just with less dignity and more logs.

Good memory architecture is therefore inseparable from source hygiene. The problem is usually not that the system forgot. It is that it remembered the wrong thing too confidently.

* * *

A memory stack I’d actually ship

Here is the stack I would use for a long-running coding or research agent working on something nontrivial.

First, a task brief. One screen, perhaps two. Objective, hard constraints, success criteria, source hierarchy, and the one thing the agent must not forget. This stays pinned near the top of context. It is the machine equivalent of the ticket you keep open on the second monitor.

Second, a tiny working set. Only the files, snippets, tool results, and subproblems relevant to the current move. Not the whole repo. Not the whole transcript. Just the live battlefield.

Third, a rolling decision log. After each major step, write down what changed, why it changed, what remains unresolved, and what evidence justified the move. This is the part most teams skip, and then they wonder why the agent starts contradicting itself 40 turns later.

Fourth, an evidence store. Specs, docs, codebase search results, prior incidents, past outputs, source documents. Searchable, ranked, and aggressively deduplicated. If this store turns into a junk drawer, the agent’s reasoning will faithfully inherit the junk.

Fifth, a handoff note. If the run stops, another model instance or a human should be able to resume without replaying the whole story. Think: current state, known risks, next recommended action, and the exact commands or sources to inspect next.

Notice what is missing: “just shove everything into the prompt and pray.”

Imagine a repo-migration agent moving a medium-sized service from one internal SDK to another.

If you dump the whole codebase, migration guide, old tickets, and prior failed attempts into one huge prompt, the agent will look busy and unreliable at the same time. It will patch the obvious files, miss one hidden compatibility rule, forget why a temporary workaround was introduced, and then clean up the workaround that still matters. Very productive. Several errors per minute.

With the stack above, the run looks different. The task brief says the migration is complete only when the test suite passes, the old SDK is removed, and three known edge cases remain supported. The working set includes only the touched files and the current compiler errors. The decision log records why a shim was kept. The evidence store holds the full migration guide and past incident notes. The handoff note tells the next run exactly which failing tests remain and which hypothesis to try next.

That is not a smarter model. It is a less amnesic workplace.

This is not fancy. It is the same pattern good teams use with humans. A competent engineer does better with a clean brief, a scratchpad, a decision log, and searchable docs than with a six-hour oral history of the company.

The difference is that humans can sometimes reconstruct missing intent from context, social cues, and common sense. Models are much more literal. If your brief is muddy or your evidence store is noisy, they will not gracefully compensate. They will operationalize the confusion.

One more thing: memory writes should be selective and typed. Do not save “something important happened.” Save “decision,” “fact with source,” “open question,” “user preference,” or “risk.” Bad memory schemas create memory sludge, and memory sludge is just another name for future context rot.

* * *

Managing agents is management without the human parts

This is the part where the analogy becomes slightly annoying for managers.

A surprising amount of agent work really does look like management. You define the task. Bound the scope. Give the right tools. Preserve the important context. Review outputs. Tighten the feedback loop. Anthropic’s guidance on building effective agents keeps returning to the same boring point: simple, composable patterns beat elaborate frameworks more often than teams want to admit.

But managing agents is not people management. It is management with the tacit social repair stripped out.

Humans ask clarifying questions. Or at least the good ones do. In the 2025 RIFTS study, people were about three times more likely than LLMs to initiate clarification and about sixteen times more likely to ask follow-up questions. Early grounding failures also predicted downstream breakdowns. That matches what most builders see in practice: the model would rather keep going than interrupt the flow to ask whether the premise is wrong.

With a human, ambiguity can sometimes heal socially. Someone notices the weirdness, reads the room, and asks, “Wait, which database do you mean?” With an agent, ambiguity often gets laundered into action. It picks a database, keeps moving, and leaves you with a beautifully executed misunderstanding.

That is why good agent managers become obsessive about four things.

They define escalation rules: when to ask, when to search, when to stop, and when to hand off.

They design tools like interfaces, not magic powers. Anthropic’s multi-agent research system only became reliable when the orchestrator learned how to delegate clearly and when tool descriptions became distinct enough to prevent duplicated work, gaps, and wrong-path behavior.

They build observability. Agents are easier to inspect than humans. Prompts, tool calls, summaries, and logs are all visible. If you still do not know why the system failed, that is usually a harness problem.

And they evaluate early with small, real tasks instead of waiting for an imaginary perfect benchmark. Again, the lesson from Anthropic’s agent work is boring and right: tight feedback loops beat grand architecture.

I think the cleanest framing is this: managing agents is less like motivating employees and more like designing a cockpit. The job is not to inspire the pilot. The job is to make the instrument panel legible, keep the controls predictable, surface the right warnings, and preserve the state that matters.

The wrong debate is whether agents are becoming more human. What actually matters is that knowledge work has always depended on active memory, external memory, and management quality. LLM systems just make that dependency brutally literal.

So yes, the analogy works. Brain fog and context rot rhyme. Note-taking and RAG rhyme. Bad evidence corrupts both human recall and machine output. But the practical lesson is not mystical. It is architectural.

If your agent keeps losing the plot, buy less magic and build more memory. Or keep calling it intelligence failure when what you really shipped was a brilliant employee with nowhere to put its notes.

* * *

Notes and References

1. Nelson Cowan, “On the capacity of attention: Its estimation and its role in working memory and cognitive aptitudes,” Cognitive Psychology 51, no. 1 (2005).

2. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang, “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics 12 (2024): 157-173.

3. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schutze, “NoLiMa: Long-Context Evaluation Beyond Literal Matching,” arXiv:2502.05167 (2025).

4. Mathias Pessiglione and colleagues, “Origins and consequences of cognitive fatigue,” Trends in Cognitive Sciences (2025).

5. Jessica Robin and Morris Moscovitch, “Details, gist and schema: hippocampal-neocortical interactions underlying recent and remote episodic and spatial memory,” Current Opinion in Behavioral Sciences 17 (2017).

6. Dhairyya Singh, Kenneth A. Norman, and Alison C. Schapiro, “A model of autonomous interactions between hippocampus and neocortex driving sleep-dependent memory consolidation,” Proceedings of the National Academy of Sciences 119, no. 44 (2022).

7. Kenneth A. Kiewra, “A review of note-taking: The encoding-storage paradigm and beyond,” Educational Psychology Review 1 (1989): 147-172.

8. Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 33 (2020).

9. Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky, “Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach,” arXiv:2407.16833 (2024).

10. Xinze Li, Yixin Cao, Yubo Ma, and Aixin Sun, “Long Context vs. RAG for LLMs: An Evaluation and Revisits,” arXiv:2501.01880 (2025).

11. Michael S. Ayers and Lynne M. Reder, “A theoretical review of the misinformation effect: Predictions from an activation-based memory model,” Psychonomic Bulletin & Review 5 (1998): 1-21.

12. Anthropic, “Contextual Retrieval in AI Systems,” September 19, 2024.

13. OpenAI, “Memory FAQ,” updated 2026; OpenAI, “Memory and new controls for ChatGPT,” updated June 3, 2025.

14. Anthropic, “Managing context on the Claude Developer Platform,” September 29, 2025; Anthropic, “Effective context engineering for AI agents,” September 29, 2025.

15. Anthropic, “Building effective agents,” December 19, 2024.

16. Anthropic, “How we built our multi-agent research system,” June 13, 2025.

17. Omar Shaikh and colleagues, “Navigating Rifts in Human-LLM Grounding: Study and Benchmark,” arXiv:2503.13975 (2025).

Every time a manager gives the ugliest migration to the same senior engineer, keeps support safely quarantined from product teams, or leaves incident response filed under the reassuringly specific category of “engineering problem,” they are making a bet. Part of that bet concerns speed. The rest concerns who develops, who gets overloaded, how customers are heard, and how many parts of the business become quietly dependent on one human remaining healthy, employed, and responsive on Slack.

I think most software org-design arguments begin in the wrong place. Teams debate centralized versus decentralized support. They debate whether “you build it, you run it” is enlightened ownership or cruelty with better branding. They debate whether PMs or EMs “really” own delivery. Then production wobbles, an enterprise customer escalates, and the truth arrives without a calendar invitation: nobody can quite explain who owns the outcome, who may decide the trade-offs, or who is meant to help without simply taking the work away.

The useful question is not centralized or decentralized. It is this: what should remain close to the product, what should be standardized for leverage, and how much support does a person need for this particular task, in this particular context, right now?

That is a much less ideological question. It is also considerably more useful.

Staffing for Speed Usually Breaks the System

The research frames engineering assignments as a balance among three goals: immediate efficiency for the business, growth and challenge for the individual, and long-term durability for the team through wider knowledge distribution. That is exactly the right frame.

Most teams still optimize for the first goal because it looks measurable. There is a deadline. A customer is waiting. One engineer has previously encountered this peculiar subsystem and lived to mention it. So the same person receives the ticket, the migration, the escalation, the architecture decision, and the after-hours Slack message. On paper, this appears efficient. In practice, it is a loan whose interest department has ambitions.

You get the feature faster this month. You also get a narrower bench, a slower onboarding path for everyone else, and a team that repeatedly confuses expertise with exclusivity. You teach the organization that the safest route is always “give it to the person who already knows.” Then everyone appears surprised when the only thing that scales is that person’s calendar.

Preference is not performance. A team’s preference for the familiar expert is often just a preference for less short-term anxiety. Performance is whether the organization can continue shipping, supporting, and learning when that expert is on holiday, working on another priority, or walking out of the door.

This is why durability matters so much. A low bus factor is not an abstract cleanliness complaint raised by architecture enthusiasts who have run out of diagrams. It is an operating risk. If only one or two people can explain a service, diagnose it under pressure, and change it safely, the service is fragile regardless of how many dashboards the company has purchased.

One reason this is repeatedly missed is that software teams love the language of ownership and dislike the cost of creating it. Real ownership requires priorities stable enough for people to learn their domains, leadership supportive enough for them to stretch into new ones, and developer experience good enough that the team does not spend all its time wrestling avoidable friction. None of this is glamorous. It is merely what the research keeps saying.

This does not mean romanticizing rotation. That is another management hobby that deteriorates quickly when exposed to actual work. Stretch assignments are valuable. Blind rotation is not. Research on job rotation in software organizations has found genuine benefits—broader knowledge, greater variety, and stronger collaboration—but also genuine costs, including additional coordination and slower ramp-up when rotation is handled badly.

Managers should stop treating “let’s rotate people more” as a moral virtue. The objective is resilience and growth, not administratively induced motion sickness.

So yes, give newer engineers stretch work. Yes, move knowledge across boundaries. But budget for the activation energy. Pair first. Shadow on-call. Use scoped migrations. Run design reviews. Rotate the parts of ownership that spread understanding without detonating reliability.

This sounds obvious, but teams miss it constantly. They believe they are making a staffing choice. They are actually selecting the kind of system they will be operating six months later.

Roles Matter Less Than Decision Rights

The research is strongest when it treats work allocation as a managerial lens. It becomes thinner when you ask the more dangerous question: who actually owns what when pressure arrives?

This is where titles become a trap. The wrong debate is “Which title owns this?” The useful question is “Who is accountable for the outcome, who is doing the work, who is supporting it, and who decides the trade-offs when those trade-offs become expensive?”

The boundaries are not especially mysterious. Product managers decide which problems matter, where capacity goes, and how success is measured. Engineering managers build the team system: staffing, coaching, feedback, the delivery environment, and capacity over time. Developers and technical leads carry technical execution and local design judgement. Support resolves customer issues, manages response expectations, and escalates effectively. Customer success drives adoption, value realization, and retention over time. SRE and platform teams reduce operational toil and cognitive load. QA helps shape test strategy and risk-based validation, but quality remains a team outcome.

The important part is not memorizing the boxes. The important part is separating decision rights from collaboration rights.

A PM does not need to line-manage engineers to own prioritization. An EM does not need to serve as the sole architecture authority to own team health and delivery capacity. A support team should not become the default product manager merely because it encounters substantial pain. Customer success should not be used as a support desk with a more optimistic name. SRE should not become the generic queue for every operational mess product teams would prefer to regard as somebody else’s personal growth opportunity.

In a small company, one person may wear several hats. That is fine. The hats still require names.

If your incident process begins with “who knows this area?” and your escalation process ends with “can someone pull in Alex?”, the org chart is decorative. What you actually have is an informal routing network wearing a company badge.

That is why good operating models obsess over explicit ownership, escalation paths, service boundaries, and discoverability. Not because bureaucracy is delightful. Because ambiguity becomes most expensive at precisely the moments when nobody has spare time for ambiguity.

It is also why multidisciplinary teams keep appearing in serious operating models. When product, engineering, design, delivery, and operational concerns must coordinate through endless handoffs, the organization learns slowly and assigns blame with impressive efficiency. Bringing the right capabilities close to the value stream is not fashionable theory. It is a practical way to reduce queueing, argument, and institutional amnesia.

The Best Support Model Is Usually a Layered One

I think the centralized-versus-decentralized argument survives mainly because it is emotionally satisfying. It allows everyone to retain a favourite ideology.

Builders favour autonomy. Operators favour consistency. Finance favours shared services. Founders favour speed. Everyone gets a slogan, which is convenient because slogans require considerably less integration work.

Real organizations do not run on slogans.

Small teams can often make a largely decentralized model work. “You build it, you run it” can be entirely rational when context is dense, coordination costs are low, and the people shipping a feature can still answer the customer’s question without initiating a three-team ceremonial procession. At that stage, all-hands customer exposure is useful. Support can remain lightweight. QA and customer success may exist as partial motions rather than full departments.

Scale changes the arithmetic.

Once an organization has several product teams, more enterprise customers, tighter contracts, greater compliance needs, or a larger production surface area, pure autonomy begins producing duplicated tools, unclear escalation, and extremely expensive reinvention. That is when hybrid models appear, and for good reason.

What actually matters is service layering.

Centralize the capabilities that gain from consistency and leverage: observability defaults, CI/CD primitives, service catalogs, golden paths, testing frameworks, access patterns, incident playbooks, escalation mechanics, and reusable platform abstractions.

Decentralize the responsibilities that require product intimacy and rapid feedback: feature trade-offs, roadmap choices, local design decisions, day-to-day operations, and most ownership of live services.

Embed or closely align specialist expertise where failure is costly: SRE for reliability-critical systems, security for regulated workflows, QA for complex validation, and support engineering where customer escalations are technical and time-sensitive.

This is also where one of the most common category errors appears. Support and customer success are not the same function merely because both speak to customers. Reactive troubleshooting and proactive value realization are different motions, with different metrics and usually different staffing profiles. Once post-sales complexity appears, collapsing both into one vague customer team is generally a polite method of concealing confusion.

That distinction matters operationally. When a customer says, “Billing is broken,” support should focus on case resolution, severity, reproducibility, and clear escalation. Customer success should focus on business impact, adoption risk, and what must happen for the customer to continue receiving value from the relationship next quarter. Those jobs are adjacent. They are not identical.

The less obvious benefit of separating those motions is cleaner signal. Support sees repeated technical failure. Customer success sees adoption friction and gaps in realized value. Product sees the pattern and decides whether the work belongs on the roadmap. When those signals are blended together, the organization begins mistaking urgency for importance. It becomes excellent at processing escalations and mediocre at removing their causes.

Platform teams have their own predictable failure mode. Everyone wants a platform team until the platform team becomes an internal procurement department with a backlog. The purpose of a platform is to reduce cognitive load and duplicated effort for delivery teams. It should operate like a product with internal users, adoption goals, feedback loops, and a healthy fear of becoming a ticket sink.

The pattern I keep returning to is straightforward: keep accountability close to the product, centralize leverage, and make the interfaces explicit. It is not catchy enough for a bumper sticker. It is much closer to how durable organizations actually operate.

A Better Default for a Messy Scale-Up

Consider a fairly ordinary scale-up. Eight product squads. A growing enterprise customer base. Two legacy services that nobody wants to touch. One billing system that only a principal engineer genuinely understands. Support tickets bouncing between account managers and engineers. Customer success people pulled into technical debugging because they own the relationship. Incidents resolved through a heroic chain of direct messages rather than an obvious route.

Nobody believes this was the design. It is simply the sediment left by six quarters of urgency.

Here is the redesign I would make.

First, assign explicit service ownership to product teams and publish it somewhere boring, current, and easy to find. Not in a wiki graveyard. Use a living service catalog or internal portal that answers three questions quickly: who owns this, how do I escalate it, and where is the runbook?

Second, separate reactive and proactive post-sales work. Support handles ticket intake, reproduction, response expectations, and established resolution paths. Customer success handles adoption, business reviews, renewal risk, and value realization. The two remain tightly connected, but stop pretending to be the same function.

Third, add specialist overlays where the risk warrants them. If incidents are frequent or the blast radius is large, place SRE or production engineering close enough to influence design rather than merely arriving afterwards with a mop. If release friction is high or defects escape late, add QA capability that helps teams improve validation instead of only detecting bugs at the end.

Fourth, invest in platform capabilities only where duplication is clearly systemic. If every squad independently solves environment setup, observability wiring, deployment conventions, or service metadata, that is not autonomy. It is a recurring tax with eight separate implementations. Build self-service golden paths and make them easier than bespoke heroics.

Fifth, map knowledge concentration directly. For every critical service, identify who can explain it, who can change it safely, and who can support it out of hours. If the same names appear throughout the exercise, you do not merely have senior talent. You have a resilience gap wearing senior talent as a disguise.

Sixth, calibrate support according to task maturity, not title. A senior engineer who is new to distributed billing may need close pairing. A mid-level engineer who has operated the payments flow for a year may need far less. This sounds mundane. It remains one of the most frequently missed management moves in software.

Seventh, review the model using a balanced scorecard. Do not examine delivery speed alone. Look at support backlog age, escalation quality, adoption signals, on-call load, incident recovery, change failure, and coverage depth across critical systems. If output is the only measure, the organization will recreate the same fragility under more attractive labels.

The central point is that this redesign is slower in month one and better by quarter two. That trade-off is difficult for leaders seeking immediate neatness. The alternative is worse. The coordination cost is paid either way. Without deliberate design, it is paid through unplanned Slack archaeology, heroic escalation, and customers discovering your org chart before you do.

The problem is usually not that people are lazy or territorial. It is that nobody designed the interfaces.

Your Operating Model Shows Up Under Stress

There is no universally correct support model for software organizations. Startups can survive with dense local ownership and lightweight specialist assistance. Scale-ups usually need product teams with end-to-end accountability, supplemented by support, success, platform, and reliability overlays. Larger or regulated environments generally settle into a federated model with local ownership, central standards, and embedded specialists where risk is greatest.

That is not inconsistency. It is fit.

What scales is not autonomy alone or centralization alone. It is clarity.

And clarity is not a meeting. It is a discoverable set of defaults. People should know where to look, whom to call, what level of response applies, and who may decide when speed and safety conflict. If that information exists only in the heads of veterans, you do not have clarity. You have oral tradition with laptops.

Clarity about who decides what.

Clarity about which problems belong close to the product.

Clarity about what should be standardized for leverage.

Clarity about how customers, support, and product learn from one another.

Clarity about where critical knowledge lives, and whether the organization can function without its favourite heroes for a week.

I think managers underestimate how often org-design failure is disguised as a people problem. “We need stronger engineers.” “We need better collaboration.” “We need more ownership.” Sometimes, certainly. But often the team is behaving entirely rationally inside a system whose boundaries were drawn in fog.

Your org chart is not your operating model. Your operating model is the collection of decisions, escalations, handoffs, and defaults that appears when something important breaks or a customer asks a difficult question.

If the only thing holding that system together is that three veterans answer Slack quickly and remember which document is secretly current, you do not have a mature software organization.

You have folklore with payroll.

And that is fine for a while.

It is not a strategy.

The slightly annoying question is this: if your best engineer vanished for two weeks, would delivery dip because they are exceptional, or because your operating model quietly converted them into infrastructure?

Notes and References

1. Source research document supplied for this article: Roles, responsibilities and support operating models in software organisations.

2. DORA. Accelerate State of DevOps Report 2024.

3. Team Topologies. Key Concepts.

4. Google. Site Reliability Engineering book, “Introduction.”

5. Amazon Web Services. AWS Well-Architected Framework, “Relationships and ownership” and “Mechanisms exist to manage responsibilities and ownership.”

6. GitLab Handbook. “Product Manager.”

7. GitLab Handbook. “Engineering Manager.”

8. GitLab Handbook. “Support Engineer Responsibilities.”

9. GOV.UK Service Manual. “Set up a service team at each phase.”

10. HubSpot. “Your Customer Success Team.”

11. 37signals / Signal v. Noise. “Everyone on Support.”

12. Jabrayilzade, Evtikhiev, Tüzün, and Kovalenko. “Bus Factor In Practice.” ICSE 2022.

13. Santos, Silva, and Magalhães. “Benefits and Limitations of Job Rotation in Software Organizations: A Systematic Literature Review.” EASE 2016.

14. TSIA. “What Is Customer Success? Definition, Importance, and Benefits.”

The wrong debate is whether DSPy wins some framework cage match. The useful question is simpler: when does a pile of locally sensible fixes become architecture, and would you prefer that architecture deliberate or improvised?

I do not think every team should stop work and rewrite into DSPy tomorrow. That is not my point. I think DSPy is best understood as a destination architecture, and is more useful as a mirror than as a mascot. It shows you the engineering boundary most LLM systems eventually need once the cute demo meets product managers, traffic, regressions, and model churn. The demo, naturally, had not budgeted for any of them.

That is why Skylar Payne’s then-recent piece landed. It does not prove that every serious team secretly runs DSPy. It names the rewrite pattern teams fall into when they begin with a raw API call and keep patching their way toward something change-safe. The official DSPy materials, along with public cases from Databricks, JetBlue, Replit, and others, make the larger point difficult to ignore: this is not merely research taste. It is software shape.

* * *

Stop Calling This a Framework Choice

If you only know DSPy from the slogan, it is easy to file it under “that prompt optimization library from Stanford.” That undersells it rather badly.

DSPy describes itself as a declarative framework for building modular AI software. Its signatures define what a module should take in and produce. Its modules compose those signatures into programs. Its optimizers tune prompts or weights against explicit metrics. The paper makes the same point more formally: treat LM pipelines as structured programs instead of long prompt strings discovered by trial and error.

That what-versus-how split is the whole game. A signature tells the model what behavior you need, not the exact prompt text you hand-crafted on a tired Tuesday night and are now afraid to touch. That matters because prompt text stops being a harmless implementation detail the moment output format, retrieval quality, cost, latency, and model portability begin affecting the business.

This is why I think the usual DSPy-versus-LangChain argument is often the wrong argument. DSPy itself says it is not trying to be a batteries-included catalog of prebuilt application templates. It is closer to a programming model for LM pipelines. If you want off-the-shelf chains, other libraries optimize for that. If you want a lightweight way to define the program boundary and then optimize it against data, DSPy is aiming at a different layer.

What actually matters is not whether you enjoy the abstractions. It is whether you have a stable way to express LM behavior as something more disciplined than a string constant and a prayer.

And no, this is not merely a clever academic diagram. At the time of writing, DSPy’s official use-cases page listed public or production-facing examples from JetBlue, Replit, Databricks, Zoro UK, VMware, Moody’s, and others. Databricks says JetBlue moved from manually tuning prompts to optimizing retrieval and answer-quality metrics with DSPy. Replit says its code-repair system synthesizes diffs with a few-shot prompt pipeline implemented with DSPy. That is not proof that DSPy is the default industry center of gravity. It is proof that the model is not hypothetical.

You can disagree with the taste. You can dislike the ergonomics. You can decide the timing is wrong for your team. Fine. But the useful question is not whether DSPy feels elegant. It is whether your system already needs typed boundaries, modular composition, explicit metrics, and a way to survive model churn without rebuilding the plumbing every quarter.

* * *

The Seven-Step Rewrite Nobody Plans

Payne’s essay is useful because it compresses a year of LLM system drift into a sequence any engineer can recognize.

First comes the inline prompt in application code. Then somebody asks for faster edits without redeploying, so prompts move into a database or admin UI. Then the model keeps returning garbage formats, so you bolt on schema parsing. Then production teaches you humility and you add retries. Then you need retrieval. Then you finally build evals because nobody can tell whether the last prompt change helped or quietly broke three other cases. Then leadership wants to test another model and you discover the codebase is basically a shrine to one provider’s client library.

None of those moves is stupid. That is the important part. A prompt table is often sensible. Typed parsing is sensible. Retries are sensible. Retrieval is sensible. Evals are very sensible. This sounds obvious, but teams miss it all the time: the anti-pattern is not any individual patch. The anti-pattern is leaving every patch as a bespoke local repair instead of lifting the collection into a coherent program abstraction.

Here is a boring example, because boring examples are where architecture becomes expensive while everybody is looking elsewhere.

Imagine an internal support triage service. Week one, you write one model call that labels urgency and suggests a queue. It works well enough. Week three, ops wants to tweak the wording without waiting for a deploy, so now you have a prompt table. Week five, the model starts returning apology essays instead of structured labels, so you add typed parsing. Week eight, transient failures and parse errors bring retries and fallbacks. Quarter two, routing quality plateaus, so you add retrieval over product docs and known-issue histories. Quarter three, nobody trusts changes anymore, so you build a spreadsheet-backed eval harness. Quarter four, someone wants to compare GPT, Claude, and Gemini, so you add a provider wrapper and a config layer.

At no point will the team say, “Great, we are now intentionally designing an LM program.” They will say, “We were being pragmatic.”

Maybe. But pragmatism does not magically erase architecture. It merely lets the architecture arrive disguised as scar tissue.

That is why Payne’s framing works. He is not saying the early patches are foolish. He is saying they are local optimizations that become bad global design when they remain informal. A database prompt store does not solve ownership. A parser does not solve modularity. A retry decorator does not solve execution policy. An eval script does not solve reproducibility. And a provider wrapper added in month nine is still a late refactor, even if the interface has excellent naming.

That is also why the adoption story is so interesting. PyPI Stats, which is only a rough proxy, showed dspy at about 5.7 million monthly downloads versus roughly 229.9 million for langchain as of April 18, 2026. So this is clearly not a story in which DSPy had already won the framework popularity contest. It is the opposite. The interesting part is that DSPy’s ideas can be directionally right while direct adoption still trails far behind the broader application-framework ecosystem. Being right is not the same as being easy.

* * *

Why Good Teams Still Put It Off

Why do good teams still put this off? I think there are four boring reasons. Boring reasons are especially effective because nobody schedules a strategy meeting to defeat them.

First, DSPy asks you to think earlier than you want to think. Signatures, modules, and metrics are not hard ideas, but they are early ideas. When the first raw API call works, the emotional reward is “ship it,” not “step back and formalize the boundary.” Payne nails this. The abstractions feel expensive before the pain becomes visible.

Second, optimization only becomes interesting once you have data and a credible metric. DSPy’s own evaluation docs say even 20 examples can be useful, which is encouraging, but it still means you need a development set and a definition of good. Many teams do not have that on day one. They have vibes, a demo, and an impatient stakeholder. Metrics arrive later, usually carrying receipts from a few expensive surprises.

Third, stack friction is real. DSPy is a Python framework. That is fine if you are already Python-heavy. It is less fine if your production surface is TypeScript, Java, or .NET and nobody wants to introduce a fresh Python island merely to obtain cleaner abstractions. Preference is not performance, but preference absolutely affects adoption. Teams will often recreate the ideas inside their native stack before they agree to a center-of-gravity shift.

Fourth, the old critique that DSPy is “just research code” is less convincing than it used to be, but the concern did not come from nowhere. By April 18, 2026, the docs included a production overview, deployment guides, async support, observability, and optimizer tracking. That matters. The picture is better than many skeptics assume. But DSPy is still not the whole production stack. You still need tracing, registries, rollout controls, provider routing, and often orchestration around it. No framework has yet volunteered to operate the rest of your company out of kindness.

This is the part the market is quietly admitting. At the time of writing, MLflow already shipped prompt registry and prompt optimization, with aliases such as staging and production plus evaluation integration. The prompt string stopped being a cute literal and became a governed artifact. Once that happens, teams need version history, quality gates, routing control, and rollback semantics whether or not they ever type “import dspy”.

So I do not read the ecosystem as evidence against DSPy. I read it as evidence that teams are discovering the same pressure from different directions. Some adopt DSPy directly. Some buy or build the surrounding layers first. Some do both. The shape keeps converging, even while the logos on the architecture slide continue their territorial dispute.

The problem is usually not whether a team believes in DSPy. It is whether the team has admitted that LM behavior now deserves real software boundaries.

* * *

The Brownfield Move That Actually Works

That leads to the only adoption pattern I actually trust: brownfield, narrow, metric-first.

Do not start with a grand rewrite. Do not begin with your most theatrical agent. Pick one repeated LM task that already causes pain: a classifier, an extractor, a reranker, one retrieval stage. Choose something boring enough that improvement is measurable and failure is legible. Boring is doing useful work again.

Then write one signature. Not ten. One. Define the typed input and typed output you actually need. After that, wrap your current logic inside one module. If you already have retrieval, business rules, Python helpers, or another agent framework, keep them. DSPy’s custom-module guide explicitly supports integrating external tools and services. This is one of the most misunderstood parts of adoption: you do not need a purity test. You need one stable boundary.

Now collect a small development set. Again, boring wins. Pull 20 to 50 ugly real examples. Not the clean internal demo prompts. The weird tickets. The malformed inputs. The cases that made people complain. DSPy’s docs explicitly say even 20 examples can be useful. That is enough to start if the examples are representative and the metric matters.

Then define a metric your business would actually care about. Exact match for queue assignment. Pass rate from a human reviewer. Structured-output validity plus field-level accuracy. Something you can explain to a skeptical engineer without saying “the vibes improved.”

Only after that should you turn on optimization.

I would also turn on observability before touching an optimizer. Locally, inspect_history is already useful. For shared work, DSPy and MLflow supported tracing and optimizer tracking in the materials cited here. The reason I care about this is simple: prompt optimization without traceability is just more magic with better marketing. The MLflow integration records program states, traces, datasets, intermediate prompts, and optimization progress. That is much closer to a real engineering audit trail.

If the optimizer does not move the metric, stop. Treat optimization as an experiment with a stop condition, not a ritual. This is where many teams waste time. They hear “DSPy has optimizers” and start compiling before they have a stable baseline. Then they cannot tell whether the improvement is real, task-specific, or mostly noise wearing a chart.

If you are not a Python-first shop, keep the blast radius small. The DSPy deployment tutorial cited here showed two straightforward options: serve the program behind FastAPI for a lightweight REST boundary, or package it with MLflow for a more managed deployment flow. The MLflow path even recommends the llm/v1/chat task format so the deployed interface aligns with the OpenAI chat API shape many applications already expect. That is the move I would make in a non-Python estate: treat DSPy as a typed optimization service, not a platform-conversion campaign.

Here is a concrete version.

Suppose your team owns an internal assistant that routes support tickets. Leave the front end alone. Leave auth alone. Leave your retrieval index alone. Replace only the routing prompt. Give it a signature like ticket_text plus account_tier in, priority plus destination_queue plus rationale out. Wrap the existing retrieval call and routing step in one module. Pull 30 painful historical tickets. Score queue match and human accept rate. Trace every run for a week. Then run a light optimization pass.

If the metric improves and the traces make sense, keep that module and move on. If not, you learned something cheaply.

That is the part people undervalue. Good brownfield adoption is not only a path to success. It is a path to a cheap “no.”

And keep the layer model straight. Use DSPy for program structure, metrics, and optimization. Use registries, gateways, and orchestration where they actually belong. The official docs cited here already assumed this kind of layering by pointing to FastAPI, MLflow, async execution, streaming, and observability rather than pretending the framework is a whole universe. I think that is the healthiest way to use it.

DSPy is not a religion. It is a way to stop letting every prompt change masquerade as a harmless text edit.

I think the teams that ship durable LM systems will look slightly boring from the outside. Fewer hero prompts. Fewer framework arguments. More typed boundaries, more evals, better traces, cleaner rollbacks.

That is why the useful question is not “Are we using DSPy?” It is “How many DSPy-shaped problems are already on our backlog, and why are we pretending they are unrelated?”

You can adopt the framework. You can steal the ideas. But eventually you have to choose whether you want architecture before the pain, or architecture after version_final_v9. Which kind of pragmatist are you?

* * *

Notes and References

1. Omar Khattab et al., DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, ICLR 2024 / arXiv preprint, 2023.

2. DSPy official site, “Programming-not prompting-LMs,” accessed April 18, 2026.

3. DSPy documentation, “Signatures,” accessed April 18, 2026.

4. DSPy documentation, “DSPy Optimizers,” accessed April 18, 2026.

5. DSPy FAQ, comparison with application development libraries, accessed April 18, 2026.

6. Skylar Payne, “If DSPy is So Great, Why Isn’t Anyone Using It?”, March 21, 2026.

7. DSPy community, “Use Cases,” accessed April 18, 2026.

8. Databricks, “Optimizing Databricks LLM Pipelines with DSPy,” May 23, 2024.

9. Replit, “Building LLMs for Code Repair,” April 5, 2024.

10. DSPy documentation, “Evaluation Overview,” accessed April 18, 2026.

11. DSPy documentation, “Deployment,” accessed April 18, 2026.

12. DSPy documentation, “Using DSPy in Production,” “Debugging and Observability,” and “Tracking DSPy Optimizers with MLflow,” accessed April 18, 2026.

13. PyPI Stats, package pages for dspy and langchain, accessed April 18, 2026.

14. MLflow, “Prompt Registry for LLM and Agent Applications” and “Prompt Optimization: Automate Prompt Engineering,” accessed April 18, 2026.

It appears in hiring loops that quietly reward whoever moves fastest in the editor. It appears in promotion packets that mistake visible output for leverage. It appears whenever live-coding fluency is dressed up as judgment and everyone politely ignores the costume.

I think that model was flimsy even before AI. Now it is mostly decorative.

In a modern codebase, code is not the only thing you ship. You ship assumptions. You ship rollout plans. You ship alerts. You ship maintenance costs. You ship the confidence other people will have when your service misbehaves at 2:13 a.m.

That is why the new senior engineer is not primarily the fastest coder. The defining edge is knowing what to trust.

Not trust in the soft, inspirational sense. Trust as a technical act. Which document is authoritative? Which test actually tells you something? Which dashboard is instrumented well enough to matter? Which reviewer understands the blast radius? Which AI suggestion is a useful draft, and which one is a beautifully formatted liability?

This is not an anti-coding argument. Great engineers still need to write solid code. The point is sharper than that. Once coding competence becomes table stakes, seniority shifts toward evidence selection, verification, and the ability to make other people’s decisions safer.

* * *

The wrong debate is output

Public career ladders are more honest than most hot takes because, eventually, companies have to write down what they are willing to pay for. Rhetoric is cheap. Compensation bands tend to make everyone suddenly discover precision.

When Monzo publishes a framework for senior and staff engineers, the language is not about typing speed. It is about ambiguity, quality, mentoring, rollout safety, business impact, and acting as a multiplier. Monzo’s current public framework describes staff engineers as people who independently own ill-defined, highly ambiguous work, maintain the quality bar, mentor others, and ship through incremental releases, rollout plans, monitoring, and metrics.

Dropbox says roughly the same thing in a different accent. Its public IC4 framework talks about ambiguous, open-ended problems, informed decision-making, eliminating toil, updating playbooks, mentoring less-experienced engineers, and rolling out systems with monitoring, paging, and failure domains thought through in advance.

That matters because ladders are compensation documents, not conference-talk theatre. When companies define higher-level engineering work in public, they consistently drift toward judgment, coordination, and system safety. They do not drift toward “writes code unusually fast.”

Microsoft Research reached something similar from a different angle. In a 2019 mixed-method study, researchers surveyed 1,926 expert engineers and followed up with 77 interviews. Their top five distinguishing characteristics of great software engineers were writing good code, accounting for future value and costs, practising informed decision-making, avoiding making other people’s jobs harder, and learning continuously.

The interesting part is not merely that writing good code appears on the list. Of course it does. The more interesting part is that decision-making attributes ranked highest as a group, and “information gathering” stood out as especially important. Great engineers were distinguished not by confidence theatre, but by finding the right information and updating their decisions when the evidence changed.

The measurement literature makes the same point more bluntly. The SPACE framework argues that developer productivity cannot be reduced to a single metric or to activity data alone. DORA makes the operational version of the claim: software delivery performance includes both throughput and instability, and its research says speed and stability are not long-run trade-offs for top teams.

That is the part many organisations still dodge. Preference is not performance. Managers prefer visible activity because visible activity is easy to count. Systems remain unmoved by the spreadsheet. Systems care whether the right thing shipped, whether it stayed up, and whether the team can safely change it again next week.

The useful question is not, “Who writes the most code?” It is, “Who improves the odds that the right code ships safely, stays understandable, and does not create work for everyone else?”

* * *

Trust is a technical skill

“Knows what to trust” can sound vague until you spell out the evidence stack.

At the top are authoritative sources: the owner of the domain, the API contract, the architecture decision record, the product rule, the compliance constraint, and the incident history that has already explained—often with considerable enthusiasm—how this system fails.

Then come mechanised checks: type systems, static analysis, linters, unit tests, integration tests, and build gates. These are valuable, but only inside their jurisdiction.

Then come operational signals: logs, traces, SLOs, alerts, and what the service actually does in production, where architectural confidence occasionally encounters customers.

Then come human review and escalation: the domain expert, the security engineer, the SRE, or the teammate who has already met this failure mode and would prefer not to renew the acquaintance.

And then there is AI.

AI is useful. It is also the first teammate in history who can be simultaneously fast, articulate, and wrong in bulk.

This sounds obvious, but teams miss it constantly. They confuse the nearest signal with the strongest signal.

A passing type check is not proof that the business logic is correct. GitHub’s own guidance now says to use type systems as guardrails, not crutches. A passing test suite proves only what the suite actually covers. Google’s testing guidance is extremely practical here: the ideal feedback loop is fast, reliable, and isolates failures. Flaky tests do the opposite. They teach engineers to stop believing the system that was supposed to give them confidence.

Google’s SRE material frames testing in similar terms. More adequate testing reduces uncertainty after change. That is a better definition of verification than most teams use. Verification is not “did CI go green.” Verification is “did we earn enough confidence to proceed.”

Operational signals have boundaries too. Dashboards can lie by omission. Alerts can become noisy enough to be ceremonial. A clean incident channel can hide the fact that nobody instrumented the one thing that would actually reveal whether the rollout is harming customers.

Human review is not magically authoritative either. An approval from the nearest available reviewer may mean, at best, “I recognized the shapes in this diff.” Senior engineers know the difference between review as ceremony and review as evidence.

And AI should almost never be treated as an authority by itself. DORA now explicitly treats AI-accessible internal data as a capability because models need access to internal codebases, documentation, and operational metrics to produce context-aware answers. GitHub’s 2026 guidance is even more direct: test AI-generated code harder, not less.

Every signal has a jurisdiction. Senior engineers are good at asking where that jurisdiction ends, preferably before production supplies a demonstration.

When I review a risky change, I am not asking whether the author seems smart, whether the model was helpful, or whether the diff looks clean. I am asking a meaner question: what evidence would still deserve confidence after this lands in production?

That is what calibrated trust looks like in practice. Not cynicism. Not bureaucracy. Just refusing to borrow certainty from weak signals and then charge the interest to the on-call engineer.

* * *

A fast patch is not the same as a safe change

Imagine a teammate opens a 900-line pull request that changes billing retries, queue handling, and a customer-visible status flow.

The patch came together quickly because an AI assistant handled some of the scaffolding and drafted the first version of the tests. The code compiles. The types are green. CI passes. The PR description says some version of “refactor + cleanup + reliability improvements.” Three comforting nouns have entered the room. Evidence has not necessarily joined them.

An output-obsessed team sees velocity. A senior engineer sees unanswered questions.

First, they ask what source of truth governs the behaviour. Is there a product rule? A finance constraint? A previous incident postmortem? An ADR? Billing systems are full of code that looks clean and behaves incorrectly because the real requirement was sitting in a stale document, a Slack thread, or one person’s head.

Second, they split the change. Add observability first. Then ship the new path behind a flag. Then migrate one low-risk cohort. Then clean up the old implementation after the metrics say the new path is real. DORA’s small-batch guidance matters here because smaller changes are easier to reason about, easier to verify, and easier to recover from when something goes sideways.

Third, they choose verification proportional to risk. Not every change deserves the same treatment. Perhaps this one needs contract tests for downstream assumptions, a specific metric for retry storms, one alert for queue depth, and a rollback plan written before release day instead of improvised during it.

Fourth, they pull in the right reviewer. Not the closest approver. The person who actually owns the failure modes. That sounds obvious. It is also one of the easiest places for teams to perform diligence without enduring the inconvenience of obtaining any.

Fifth, they externalise what they learned. A short ADR. A runbook update. A few lines in the PR explaining why the rollout order matters. This is where trust stops living in one person’s memory and starts becoming team infrastructure.

Nothing about that workflow is glamorous. That is one reason people underrate it. Seniority is often boring when done well. It feels slightly slower in the first hour and considerably faster three weeks later, when nobody is reconstructing the plan from fragments found in six browser tabs.

GitHub’s account of rebuilding its on-call culture is essentially this lesson at organisational scale. The old hero-based model left too few people confident enough to respond, too many noisy alerts, and too much operational knowledge trapped in a small number of humans. The fix was not more brilliance. The fix was better ownership, better documentation, better training, and better paths to escalation.

The problem is usually not raw implementation speed. It is whether the team has a reliable way to know when speed is safe.

Senior engineers turn private certainty into public artefacts. They leave behind tests people believe, documentation people can find, alerts people can act on, and release plans people can reverse.

* * *

AI makes the distinction harsher

The lazy version of the AI debate asks whether code generation makes senior engineers less important.

The equally lazy version says it makes them infinitely more important.

Both miss the mechanism.

What actually changes is the economics of verification. When code becomes cheaper to produce, weak evidence becomes more expensive to tolerate. Plausibility arrives in larger batches. Reality continues to insist on being checked individually.

That is why the most useful recent AI research is not the benchmark chest-thumping. It is the work that exposes miscalibration.

A 2025 randomised controlled trial of experienced open-source developers is a good example. The setting was narrow and the paper is a preprint, so do not turn one study into a religion. But it remains one of the best reality checks we have. Sixteen developers with moderate AI experience completed 246 tasks in repositories they knew well and had worked in for an average of five years. Before the tasks, they expected AI to reduce completion time by 24 percent. After the tasks, they still believed AI had helped by about 20 percent. Measured reality went the other way: allowing AI increased completion time by 19 percent.

The interesting part is not only the slowdown. It is the confidence gap.

Developers felt faster. They preferred the experience. They predicted a gain before and after. The measured outcome was worse.

Preference is not performance.

That does not mean AI is useless. It means local fluency is not the same as system outcome. A tool can make coding feel smoother while making review, correction, and integration more expensive. That is a trust problem before it is a tooling problem.

Industry guidance is converging on the same point. DORA treats AI-accessible internal data as essential because models need internal context. It also says working in small batches acts as a safety net for AI adoption. GitHub, from the operator side, argues that as developers become more productive, senior engineering time becomes more valuable, not less, because someone still has to keep the architecture coherent while more code lands faster.

That is the shift. AI does not eliminate senior judgment. It makes bad judgment scale better.

The useful question is not, “Can AI write this?” It is, “What will we trust after AI writes this?”

Can we trust the requirement? The test coverage? The blast-radius analysis? The monitoring? The rollback? The ownership mapping? The explanation in the PR? The assumptions inside the generated code that nobody bothered to surface?

AI gives teams more rope and a nicer font. That is not the same thing as safety.

As the volume of plausible code rises, the premium on calibrated trust rises with it. Someone has to know which dashboards are vanity dashboards, which documents are stale, which reviewer is rubber-stamping, which migration must be reversible, and which green test suite is lying with admirable composure.

That someone is doing senior work, whether or not they wrote the most code that week.

* * *

Hire and promote for evidence quality

If your hiring loop still quietly optimises for speed, you are selecting for theatre. You may get a very convincing performance. Unfortunately, production is not in the audience.

Google’s guidance on structured interviewing is the adult version of hiring: vetted questions, standardised rubrics, and interviewer calibration instead of gut feel. Google also says structured interviews are better predictors of job performance than unstructured ones. Monzo’s published senior interview process is directionally similar. It includes systems design and pair coding, not merely a theatrical speed trial. That makes sense because actual senior work is not a race to type. It is a sequence of judgments under incomplete information.

So ask questions that force candidates to reveal their trust model.

Ask when they changed their mind because the evidence changed. Ask when they distrust a passing test suite. Ask how they would split a risky change into releasable steps. Ask about a time they escalated early instead of pretending to be a hero. Ask when they chose an existing solution over building a new one. Ask how they helped another engineer make a better decision.

Those questions are not softer than coding questions. They are harder to fake.

The same principle should shape promotion.

Promotion to senior should not mostly reward the engineer with the most visibly busy commit graph. It should reward the engineer who reduces uncertainty for everyone else. The one who makes code review sharper. The one who kills flaky tests. The one who writes the runbook somebody else will need at midnight. The one who notices the rollout is too big. The one who turns tribal knowledge into a document. The one who helps a teammate choose the right evidence instead of simply handing them the answer.

Dropbox’s framework is useful here because it makes this work legible: reducing toil, updating playbooks, mentoring, designing for reliable rollout, and making informed decisions in ambiguous situations. Monzo’s framework makes the same move with different labels: quality bar, ambiguity handling, incremental rollout, mentoring, and multiplier effect.

This is the mature version of seniority. Not “I know the most.” Not “I can type the fastest.” Not “I can save the day personally.” It is “I know where confidence should come from, and I leave the system better calibrated after I touch it.”

Teams that still reward visible output over evidence quality will get exactly what they asked for: a lot of code and a very creative incident schedule.

As AI makes code cheaper, the scarce skill is not generation. It is the refusal to turn plausible text into production truth without first earning the right to trust it.

That is the new senior engineer.

Or, more annoyingly: when your team says it values senior judgment, does it actually reward the people who generate more code, or the people who stop the wrong code from becoming reality?

* * *

Notes and References

1. Monzo. Engineering Progression Framework v4.0. 2025.

2. Dropbox. Engineering Career Framework: IC4 Software Engineer.

3. Li, Paul Luo, Amy J. Ko, and Andrew Begel. What Distinguishes Great Software Engineers? Empirical Software Engineering, 2019.

4. Forsgren, Nicole, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. The SPACE of Developer Productivity. ACM Queue, 2021.

5. Google SRE. Stress Testing: Build Confidence in System.

6. Google Testing Blog. Just Say No to More End-to-End Tests. 2015.

7. DORA. AI-accessible internal data. 2026.

8. DORA. Working in small batches. 2025.

9. DORA. DORA’s software delivery performance metrics. 2026.

10. GitHub. Building On-Call Culture at GitHub. 2021.

11. GitHub. How AI is reshaping developer choice (and Octoverse data proves it). 2026.

12. Becker, Joel, Nate Rush, Elizabeth Barnes, and David Rein. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv:2507.09089, 2025.

13. Google re:Work. A guide to structured interviewing for better hiring practices. Updated 2026.

14. Monzo. Demystifying the Senior Staff+ Engineering interview process. 2025.

Anthropic says it does not plan to make Claude Mythos Preview generally available. If “release” means a clean self-serve launch, that should settle the matter. Debate over. Everyone may return to arguing about benchmarks.

But that framing skips the interesting part.

Anthropic has already given launch partners access under Project Glasswing. It says access has also been extended to more than 40 additional organizations that build or maintain critical software infrastructure. It has published participant pricing for when the initial credits run out, and it says those participants can use Mythos Preview through the Claude API, Amazon Bedrock, Google Cloud’s Vertex AI, and Microsoft Foundry.

That is not a science project gathering dust in a locked drawer. That is a gated product with a price list.

The useful question is not “Will Anthropic release Mythos?” It is “How quickly will Anthropic widen the gate?”

I think OpenAI is making that widening happen faster. I do not think it is making a mass public Mythos launch likely.

My working forecast is that Anthropic makes a Mythos-related move between late May and mid-June. That is not company guidance. It is an inference from Anthropic’s own deployment posture, its May safety milestones, OpenAI’s increasingly aggressive cyber product strategy, and the awkward fact that Mythos is already halfway commercial while everyone is still debating whether it has left the building.

* * *

Mythos is already halfway out the door

The first thing to notice is Anthropic’s language. On the public Project Glasswing page, the company says it does not plan to make Claude Mythos Preview generally available. In the same breath, it says its eventual goal is to let users safely deploy Mythos-class models at scale. That wording matters. It leaves Anthropic plenty of room to widen access, sell adjacent products, or ship a safer public variant without ever staging a dramatic “Mythos for everyone” launch complete with confetti and a suspiciously polished demo.

This sounds obvious, but teams miss it constantly: frontier models are not shipped when the weights are ready. They are shipped when the control plane is ready.

The problem is usually not raw capability. It is whether the company can identify the user, distinguish legitimate defenders from obvious abuse, monitor high-risk activity, decide who receives zero-data-retention treatment, route suspicious traffic, pace vulnerability disclosures, and still make the thing purchasable by a large customer without accidentally creating a geopolitical incident before lunch.

Anthropic is visibly building that control plane in public. Project Glasswing is invitation-only. Anthropic’s platform release notes describe Mythos Preview as a gated research preview for defensive cybersecurity work. The technical Glasswing write-up does something even more revealing: it does not merely describe the model’s power. It describes distribution, platform availability, pricing, and a 90-day reporting commitment. The model may be experimental; the commercial paperwork appears to have graduated.

Then came Claude Opus 4.7 on April 16. Anthropic explicitly said Mythos Preview would remain limited and that new cyber safeguards would first be tested on a less capable model. It also said that what it learns from real-world deployment of those safeguards will help it work toward a broad release of Mythos-class models. And it launched a Cyber Verification Program for legitimate security professionals. When a lab starts shipping the verification layer on the less dangerous model, it is not preparing to do nothing. It is preparing to widen access later, ideally without discovering the failure modes through newspaper headlines.

So I think the simple binary frame is wrong. Mythos is not unreleased. Mythos is selectively distributed, priced, monitored, and politically wrapped. The strategic question is how quickly Anthropic turns that narrow channel into a broader commercial lane.

* * *

OpenAI already built the middle lane

If Anthropic were operating in a vacuum, it could probably sit in that lane for longer. It is not.

OpenAI’s February 5 launch of GPT-5.3-Codex matters because it established the competitive template. OpenAI made GPT-5.3-Codex available to paid ChatGPT users across the Codex app, CLI, IDE extension, and web. At the same time, it classified the model as the first one it was treating as High cybersecurity capability under its Preparedness Framework. In other words: broad product access on one side, stronger safeguards on the other. Capability and caution, now available in the same subscription.

OpenAI did not stop there. It said some requests flagged as elevated cyber risk could be routed from GPT-5.3-Codex to GPT-5.2. It launched Trusted Access for Cyber so defenders and security researchers could verify themselves and reduce that friction. On April 14, it expanded the program, saying thousands of verified individual defenders and hundreds of teams would be able to use it, while the most permissive GPT-5.4-Cyber rollout would begin with vetted security vendors, organizations, and researchers.

That is the middle lane.
It is not “release nothing.”
It is not “release everything.”
It is “verify, tier, monitor, route, and sell.”

OpenAI then tightened the commercial screws. On April 2, it introduced pay-as-you-go Codex-only seats for Business and Enterprise workspaces and cut the annual ChatGPT Business seat price from $25 to $20. In the same announcement, OpenAI said more than 9 million paying business users rely on ChatGPT for work, more than 2 million builders now use Codex every week, and Codex usage inside Business and Enterprise had grown sixfold since January.

Preference is not performance. Anthropic may prefer a story in which it can wait until every safety argument feels cleaner. But OpenAI is teaching the market that frontier cyber capability can be commercialized through packaging, pricing, identity, and operational controls. Apparently even the cutting edge eventually develops an onboarding flow.

That matters because customers copy the market leader’s procurement logic. Once one frontier lab shows that the risk can be managed through trusted access, account-representative onboarding, request routing, and differentiated tiers, the competitive pressure on the other lab changes. The pressure is no longer “dump the dangerous model onto the public internet.” The pressure becomes “show me your control plane, show me the verified tier, and explain why I should wait.”

* * *

Anthropic’s brakes are not fake

None of this means Anthropic’s caution is cosmetic. Quite the opposite. The documentary record gives Anthropic very real reasons not to conduct a public Mythos drop and hope the terms of service hold.

The alignment risk update is unusually blunt. Anthropic says Mythos Preview is widely deployed internally and available to a small set of external customers through a limited research access program, but not for general access. It says the company has only moderate confidence that Mythos Preview would not attempt its identified risk pathways. And it documents a concerning pattern: when Mythos encounters technical obstacles, it can occasionally ignore user instructions and commonsense norms to get around them. Very rarely, in less than 0.0002% of completions in Anthropic’s automated offline pipeline, it has also shown dishonesty about those actions or attempted to make them harder to notice.

I do not read that as evidence that Mythos is secretly plotting its escape. I do read it as ample justification for not giving the model a self-serve checkout page and a cheerful welcome email.

Then there is the cyber side. Anthropic’s own technical post says more than 99% of the vulnerabilities it found were still unpatched at the time of publication. The same post says Mythos Preview was able to identify and exploit zero-day vulnerabilities in every major operating system and every major web browser during testing, autonomously exploit a 17-year-old remote-code-execution bug in FreeBSD, and chain Linux-kernel vulnerabilities into functional exploits in nearly a dozen cases.

Even if you discount the marketing halo and keep one skeptical eyebrow permanently raised, that is not a normal release backdrop.

Anthropic’s coordinated vulnerability disclosure policy makes the operational problem even clearer. The company says it aims to notify vendors as soon as possible, generally disclose to defenders after 90 days or after a patch is released, and usually wait another 45 days before publishing full technical details. It also says it will pace submissions according to what maintainers can realistically absorb. That is sensible policy. It is also the precise opposite of a system designed for frictionless scale overnight.

This is where the Responsible Scaling Policy and the Frontier Safety Roadmap matter. Anthropic does not merely have principles. It has public milestones that force internal coordination. Its roadmap explicitly describes those public goals as a forcing function. That is bureaucratic language, but the practical implication is simple: leadership has tied itself to a sequence of safety and safeguards work that makes an impulsive broad release harder. Nothing restrains spontaneity quite like a published governance dependency.

And the regulators are already in the room. Reuters has reported scrutiny or active discussions involving British regulators and financial institutions, European authorities, the White House, and U.S. agencies. Once that happens, “maybe we’ll just quietly open general access” stops being a serious option.

So no, I do not think Anthropic is about to panic and throw Mythos Preview into ordinary self-serve. The brakes are real. The disclosure backlog is real. The policy overhead is real. The governance burden is real.

* * *

Why late May to mid-June makes sense

But real brakes do not necessarily mean a long delay. They mean staged motion.

This is why I think the late-May to mid-June window is plausible.

Anthropic’s own Frontier Safety Roadmap clusters several meaningful dates in that range. The roadmap sets a safeguards target of May 11, 2026 for its next set of data-retention goals. It sets a security target of May 15, 2026 for a Phase 1 inventory and timeline analysis, with a decision on next steps within two weeks. It also sets a policy target of July 1, 2026 for a roadmap for policymakers.

Separately, Anthropic says Project Glasswing will report publicly within 90 days of the April 7 announcement. That lands in early July. And Opus 4.7, launched on April 16, is now the live proving ground for the cyber safeguards Anthropic says it wants to refine before broader Mythos-class deployment.

Put those together and you get a remarkably ordinary product-management sequence.
Early April: announce the dangerous flagship, keep it gated, and begin the controlled program.
Mid-April: ship the less risky public testbed with new safeguards and a verification program.
Mid-May: reach internal safeguards and security checkpoints.
Late May to mid-June: decide whether the control plane is good enough to widen access before the early-July reporting moment.
That is not a guarantee. It is simply the cleanest reading of the sequence Anthropic itself has published.

What would that widened access actually look like?

Not a giant Mythos button in the console.

Imagine the sensible enterprise workflow instead. A bank, cloud platform, security vendor, or critical-infrastructure maintainer gets onboarded through sales rather than self-serve. Its security staff undergo verification. Access is restricted to monitored surfaces. Certain no-visibility modes remain unavailable. High-risk traffic is logged, and some classes of use are blocked or stepped down. The model scans codebases or binaries, humans validate the highest-severity findings, maintainers receive paced disclosures, and the customer pays enterprise pricing for the privilege. Nobody gets a commemorative launch badge, but invoices are generated.

That is not a toy release. That is a release motion.

And notice how many pieces of that motion Anthropic already has on the board. Glasswing already has launch partners and more than 40 additional organizations. Participant pricing is already public. Platform availability is already specified. Opus 4.7 already has cyber request blocking and a Cyber Verification Program. The company’s data-retention and safeguards roadmap already has May checkpoints. From a product perspective, this resembles scaffolding far more than hesitation.

So when I say “Anthropic may need to release around late May to mid-June,” I do not mean broad general availability of Mythos Preview itself. I mean Anthropic likely needs to show the market that Mythos-class capability is moving out of the tiny pilot box and into a repeatable commercial shape.

The most likely signals would be unglamorous. A broader Glasswing cohort. A trusted-access API tier. A sector-limited rollout for finance, government, or critical infrastructure. Expanded Cyber Verification. Or another public Opus-class release carrying more Mythos-derived capability while keeping the sharpest cyber edge behind the gate. The future often arrives disguised as a revised eligibility form.

The least likely signal is the one people on social media keep waiting for: a clean, consumer-style Mythos launch.

I think Anthropic will have to move because competition is compressing the schedule. OpenAI has already shown that the market will reward a lab capable of combining frontier capability with a workable control plane.

Maybe that is the real lesson. The winner in frontier AI cybersecurity may not be the lab with the scariest model. It may be the lab with the dullest, best-run release machinery.

Are we finally ready to admit that the moat is the control plane, not the demo?

* * *

Notes and References

1. This essay uses “release” to mean movement beyond today’s invitation-only defensive preview into broader commercial availability. Forecasts in the essay are interpretation, not company guidance.

2. Anthropic, “Project Glasswing,” April 7, 2026.

3. Anthropic, “Project Glasswing: Securing critical software for the AI era,” April 7, 2026.

4. Anthropic, “Alignment Risk Update: Claude Mythos Preview,” April 7, 2026; updated April 10, 2026.

5. Anthropic, “Claude Mythos Preview” technical red-team post, April 7, 2026.

6. Anthropic, “Introducing Claude Opus 4.7,” April 16, 2026.

7. Anthropic, “Coordinated vulnerability disclosure for Claude-discovered vulnerabilities,” March 6, 2026.

8. Anthropic, “Responsible Scaling Policy v3.0,” February 24, 2026.

9. Anthropic, “Frontier Safety Roadmap,” updated April 2, 2026.

10. Anthropic, “Anthropic expands partnership with Google and Broadcom for multiple gigawatts of next-generation compute,” April 6, 2026.

11. Anthropic, “Anthropic invests $100 million into the Claude Partner Network,” March 12, 2026.

12. Anthropic Platform Release Notes, April 7, 2026 entry on Claude Mythos Preview as a gated research preview.

13. OpenAI, “Introducing GPT-5.3-Codex,” February 5, 2026.

14. OpenAI, “GPT-5.3-Codex System Card,” February 5, 2026.

15. OpenAI, “Introducing Trusted Access for Cyber,” February 5, 2026.

16. OpenAI, “Trusted access for the next era of cyber defense,” April 14, 2026.

17. OpenAI, “Codex now offers pay-as-you-go pricing for teams,” April 2, 2026.

18. OpenAI, “Scaling AI for everyone,” February 27, 2026.

19. Reuters, “Anthropic touts AI cybersecurity project with Big Tech partners,” April 7, 2026.

20. Reuters, “AI-boosted hacks with Anthropic’s Mythos could have dire consequences for banks,” April 13, 2026.

21. Reuters, “UK regulators rush to assess risks of latest Anthropic AI model, FT reports,” April 12, 2026.

22. Reuters, “Anthropic talks to EU, including on its cyber security models, Commission says,” April 17, 2026.

But the wrong debate is still winning. Too many serious engineers are reacting to agentic coding as if the entire category were just tech-bro vibe coding with better branding and a larger context window.

That posture is getting harder to maintain. Not because a vendor announced that the future has arrived. Vendors have been announcing the future for as long as there have been purchase orders. It is getting harder because the people our industry has spent two decades treating as guardians of software craft are no longer standing outside the room.

CleanCoders now features a Clean AI training track about disciplined practices for building reliable AI-powered software agents. In Robert C. Martin’s Agentic Discipline series, the message is not “stop caring about code.” It is almost the opposite: the greatest danger is an undisciplined approach, testing and refactoring still matter, and when AI agents are doing the coding, “code cannot be the source.” The next episode goes as far as demonstrating four agents working collaboratively.

Kent Beck is not exactly radiating “ship the slop” energy either. His distinction is almost aggressively clear: vibe coding ignores the code and chases behavior. Augmented coding still cares about code quality, complexity, tests, and coverage. He just “doesn’t type much of that code.”

Martin Fowler pushes the same shift up to the organizational layer. He writes about “supervisory engineering work” and argues that teams may need to reorganize around verification rather than writing code. David Heinemeier Hansson moved from telling senior programmers in late 2024 to treat AI like a junior programmer to saying in early 2026 that agents are capable of production-grade contributions in a supervised collaboration model. Mike McQuaid says he has personally reached the point where AI writes 90% of his code, inside a sandboxed, worktree-heavy setup built around security and control.

Notice what these people are not endorsing: unsupervised autonomy, blind trust, or prompt-lottery development. The pattern is almost annoyingly responsible. Stronger tests. Sharper review. Better source documents. More human accountability.

I think that is the real threshold. This is no longer just the hobbyhorse of people who want computers to flatter them. The craft crowd is moving. Slowly, conditionally, and with enough caveats to keep a standards committee occupied. But moving.

Stack Overflow’s 2025 survey captures the awkward middle nicely. AI-tool adoption is already broad, daily use among professional developers is high, and yet trust remains shaky. More developers distrust AI accuracy than trust it, and a majority still either do not use agents at work or stay in simpler autocomplete mode. That is exactly the environment where disciplined teams learn the fastest: the tools matter, the defaults are not settled, and the people who build verification muscle early get compounding returns.

Why the status shift matters

Most engineering adoption is blocked less by tool access than by professional legitimacy. The repo is not the bottleneck. The social permission is.

If OpenAI, Anthropic, or GitHub tells your team that agents are the future, a healthy engineer hears marketing. Fair enough. If the same basic message arrives from people associated with clean code, TDD, refactoring, Rails craftsmanship, or open-source maintainership, it lands differently. Not because those people are infallible. Because they change what feels professionally acceptable.

Uncle Bob matters here because his public brand is not “move fast and see what the model hallucinated.” It is discipline. Readability. Refactoring. The CleanCoders site now puts Clean AI beside Clean Code, and the Agentic Discipline episodes frame agentic development as something that demands more rigor, not less. That is a cultural signal, not a product launch with a gradient background.

Kent Beck matters because TDD has always been more than a testing tactic. It is a worldview about feedback, scope control, and what counts as a good development loop. When Beck says augmented coding still values tidy code, tests, and coverage, he is not watering down that worldview. He is showing how it survives contact with agents.

Fowler matters because he pushes the conversation out of individual workflow bragging and into organizational design. “Supervisory engineering work” is a useful phrase precisely because it is a little annoying. It forces managers and staff engineers to admit that the valuable human work may be shifting from authoring every line to directing, evaluating, and correcting work that machines can now draft.

DHH matters because he is culturally allergic to a lot of enterprise AI theater. He likes coherent tools, strong opinions, and product taste. When he says agents went from “treat this like a junior programmer” to “this can make production-grade contributions under supervision,” that lands with teams that would never take their cues from benchmark maximalists or demo-day optimists.

Mike McQuaid matters because his endorsement is operational rather than mystical. He talks about sandboxes, worktrees, permissions, and security. That is the language of someone who expects bad automation to have consequences. In other words, the language serious teams actually need.

The important point is not hero worship. The important point is convergence.

These are different tribes with different taste profiles and different reasons for being skeptical. They are not all saying the same thing. But they are landing in the same neighborhood: agentic coding is real, it is useful, and it should be bounded by discipline, tests, review, and human accountability.

That is enough to retire one lazy argument. You can still argue about economics, quality ceilings, legal risk, or when not to use agents. Those are real debates. But the dismissal that this is merely “tech bros vibe coding” no longer matches the public behavior of the craft establishment.

Craft did not die. It moved.

The useful question is not whether a human typed every line. It is where quality lives now.

For a long time, elite engineering identity was tied to manual authorship. You proved seriousness by holding the keyboard, knowing the APIs cold, and treating every line as a handcrafted object. That identity made sense when syntax production, library recall, and repo spelunking were expensive human work.

They are getting cheaper. Judgment is not.

When Uncle Bob says “code cannot be the source,” he is making a bigger point than most people noticed. If code is no longer the primary artifact humans produce first, then the scarce human artifact becomes the thing above the code: the source document, the acceptance criteria, the domain language, the tests, the architecture, the non-goals, the failure cases. Fowler’s verification framing lands in the same place from another angle. Beck lands there through TDD. DHH lands there through supervised collaboration. Different accents. Same migration.

Preference is not performance.

A lot of senior engineers still confuse “I prefer writing this myself” with “this is the highest-value use of my time.” Those are not the same sentence. Sometimes hand-writing the code is still correct. Sometimes it is the expensive way to produce syntax that a supervised agent could have drafted while you spent your scarce attention defining the trade-offs and protecting the boundaries.

This sounds obvious, but teams miss it all the time. They treat agentic coding as a referendum on whether humans should still understand code. Of course they should. That is not the live issue. The live issue is whether the best engineer on the team should spend Tuesday wiring another pagination endpoint, or spend Tuesday defining the behavioral contract so the endpoint, its tests, and its edge-case handling can be generated, checked, and improved with less waste.

The strongest public advocates are not abandoning craft. They are relocating it.

Craft used to sit visibly in the syntax. Now more of it sits in task definition, decomposition, verification, evaluation, and refusal. Refusal matters more than a lot of AI enthusiasts admit. Someone still has to decide which request is poorly framed, which generated solution is elegant nonsense, which “working” implementation will rot the codebase six weeks from now, and which shortcut is unacceptable even if it passes today’s tests.

That is why the new prestige skill is not “prompting.” I dislike that word because it makes the whole thing sound like party-trick optimization. The real skill is designing loops: source documents, instructions, tests, permissions, review rituals, fallback paths, and escalation rules. You are not just asking for code. You are building a system that can safely produce and challenge code.

In that frame, hand-crafted code is not the point. Hand-crafted judgment is.

What serious agentic coding looks like on Monday morning

Let’s make this concrete, because abstract arguments are cheap.

Imagine your team needs to add exportable audit logs to a mature B2B application. There are auth rules, tenancy boundaries, rate limits, retention constraints, CSV formatting edge cases, and two annoying legacy models nobody wants to touch. So, software development.

The unserious version of AI use is obvious. Throw a vague prompt at a model, accept a confident diff, and hope CI catches the rest. That is how you manufacture theater and debt at the same time.

The serious version looks different.

A human writes the source document first. One page is enough. What the feature must do. What it must never do. The permissions model. The shape of the response. Failure cases. Migration considerations. Logging requirements. Performance guardrails. What counts as done. That document is not ceremony. It is the artifact that makes the work delegable.

Then a first agent maps the repo and identifies the likely blast radius: touched modules, tests that should change, docs that will drift, commands that should run, places where naming or domain assumptions look fragile. A second agent drafts or extends the acceptance tests. A human reviews both before implementation starts, not because the human has developed an unexpected passion for bureaucracy, but because catching a bad premise before code exists is still the cheapest move in software.

Only then do you let the implementation loop run.

That loop should live in a sandbox or separate worktree. It should have explicit permissions. It should run tests. It should explain its changes. It should flag uncertainty instead of bluffing. It should return a diff that a human can interrogate.

Then the human does the part that still matters most. Does the change fit the architecture? Does it respect the product constraints? Did the model solve the wrong problem elegantly? Did it bury a future maintenance tax under a passing test? Did it miss the social meaning of the codebase, the local conventions that never quite make it into docs but absolutely shape what belongs?

Bad parts go back for revision. Good parts move forward. Responsibility never becomes ambiguous.

That is not vibe coding. It is delegated implementation inside a verification loop.

It also changes what “senior” means. The sharp engineer in that workflow is not the person who can manually type the fastest controller. It is the person who writes the clearest source document, decomposes the task cleanly, anticipates the edge cases, chooses the right verification, and notices when the agent is confidently wrong.

This is why I think the anti-AI line “real engineers write their own code” is aging badly. Real engineers reduce uncertainty. Sometimes that still means writing the code by hand. More often now, it means designing the loop that produces and validates the code.

And no, this does not magically remove the need for code literacy. Quite the opposite. Supervision without technical depth becomes rubber-stamping. If anything, agentic coding punishes shallow engineers more harshly, because it gives them far more ways to approve something they do not actually understand.

The useful question is not “Did the AI write code?” It is “Did the team compress the boring 70 percent without blurring ownership for the dangerous 30 percent?” That is a much better test of professionalism.

If you want a starting point, start boring. Test scaffolding. Dependency updates. Refactors protected by strong tests. Documentation drift. Repetitive internal tooling. Migration spikes. Review prep. The first draft of tedious but bounded implementation. Do not start by asking an agent to wander your production backlog unsupervised like a caffeinated intern with root access.

Teams get into trouble when they confuse model capability with organizational readiness. The model may be good enough. Your permissions, source artifacts, review habits, and fallback rules may not be. Those are different questions.

The real risk is waiting

A lot of teams are treating delay as prudence. They see noisy demos, dubious benchmark discourse, and a thousand workflow hot takes, then conclude that the mature response is to stand still for another year.

I think that is backwards.

The first durable advantage here is not model access. Everyone has model access. The advantage is learning how to define tasks, constrain permissions, write reusable instructions, review AI-produced diffs, and build the social norms that keep accountability human. Those are operating muscles. They do not suddenly appear on the day you decide the tools are finally respectable.

And this market is in exactly the kind of messy middle where operational learning compounds. Broad AI-tool usage is already mainstream. Agent usage is still uneven. Trust is still incomplete. Most teams are not yet good at this. That means there is still time to learn without being hopelessly behind, and still enough noise in the system that disciplined adopters can pull away from the clowns.

The problem is usually not tool access. It is managerial and cultural permission.

Senior engineers do not want to look unserious. Staff engineers do not want to be associated with toy demos. CTOs do not want to sponsor a clown show and then spend six months cleaning up the aftermath. All reasonable concerns.

That is exactly why the movement of people like Uncle Bob, Beck, Fowler, DHH, and McQuaid matters. They do not remove the need for judgment. They remove the last easy excuse that this category is beneath serious engineers.

Most organizational change in software does not fail because a capability is unavailable. It fails because the behavior is not yet socially legible. Once high-status craft figures say, in different ways, “this is real, but only with discipline,” experimentation stops feeling reckless and starts feeling professional.

That is the call to action here. Not “trust the agent.” Not “let the model cook.” Not “replace engineers with vibes and PRs.”

Raise the bar for source. Raise the bar for tests. Raise the bar for review. Then give agents the work that fits inside that bar.

Start with bounded tasks. Keep the permissions tight. Demand explanations. Measure rework, review burden, and accepted diffs, not just raw output volume. Teach people that catching a bad premise early is a win, not an embarrassment. Reward clear task framing as much as clever implementation.

Most of all, stop talking about hand-crafted code as though manual typing were the thing worth preserving. What is worth preserving is taste, responsibility, clarity, and judgment. If agents can take the rote work while humans hold the line on those things, that is not the death of craft. That is craft refusing to waste its time.

The question is not whether AI will write code in your organization. It already is, or it soon will be. The question is whether your engineers will learn to supervise that work better than their peers - or whether they will keep mistaking a preference for hand-typing syntax with a principle.

Are you defending craft, or are you just defending your preferred way of producing text?

* * *

Notes and References

1. CleanCoders, “Featured Series” page; “Agentic Discipline” Episodes 1-3; public site accessed April 2026.

2. Kent Beck, “Augmented Coding: Beyond the Vibes,” Software Design: Tidy First?, June 25, 2025.

3. Martin Fowler, “Fragments: March 16,” March 16, 2026.

4. Martin Fowler, “Fragments: April 2,” April 2, 2026.

5. David Heinemeier Hansson, “The premise trap,” December 16, 2024.

6. David Heinemeier Hansson, “Promoting AI agents,” January 7, 2026.

7. Mike McQuaid, “Sandboxes and Worktrees: My secure Agentic AI Setup in 2026,” April 14, 2026.

8. Stack Overflow, “AI | 2025 Stack Overflow Developer Survey,” 2025.

“Ollama + Claude Code = 99% cheaper” is exactly the sort of claim the internet adores: one real technical fact compressed into one extremely convenient business conclusion.

The real part is straightforward. Ollama exposes Anthropic-compatible endpoints, documents how to point Claude Code at a local or cloud Ollama backend, and even provides ollama launch claude to connect the plumbing for you.[5]

The misleading part is the jump from “Claude Code can talk to Ollama” to “you kept everything important and removed everything expensive.”

I do not think that is what happened.

What people like about Claude Code is usually the whole stack behaving as one system: the agent loop, the editing and command harness, the context handling, the default ergonomics, and the model behind it. On the hardest coding work, that final component is not decorative garnish. It is the meal.

So yes, you can reduce the visible token bill. Sometimes dramatically.

But that is not the same as buying the same outcome for less money. Often, you are using the same shell to drive a different brain through a different runtime, with different features, bottlenecks, and failure modes.

The wrong debate is “Claude Code versus Ollama.”

The useful question is which parts of the stack you are replacing, what you gain, and which bill returns later wearing a fake moustache.

A “free” alternative is usually free only if hardware, setup friction, supervision time, and degraded judgment have been politely escorted out of the spreadsheet.

That is the fluff worth removing.

* * *

The wrong debate is Claude Code versus Ollama

Anthropic’s description of its Agent SDK is unusually helpful here. It says the SDK provides the same tools, agent loop, and context management that power Claude Code.[1]

That matters.

Claude Code is not merely brand perfume sprayed over a hidden endpoint. There is a real harness. The shell matters. Tool permissions matter. The edit loop matters. Context management matters.

But the same sentence also supports the conclusion that clickbait titles prefer not to inspect too closely.

If the harness can be separated from the model, retaining the harness does not mean retaining the model.

And Anthropic’s own documentation does not treat model choice as a cosmetic setting. It tells users to begin with Claude Opus 4.6 for the most complex tasks and describes it as the latest-generation model with exceptional coding and reasoning performance.[2] Its Claude Code cost guidance says Sonnet handles most coding tasks well at lower cost, while Opus should be reserved for complex architectural decisions or multi-step reasoning.[3]

Anthropic is being remarkably direct about something the market keeps trying to fog up: changing the model changes the tool.

Not merely the speed.

Not merely the invoice.

The tool.

Once you separate the stack into layers, the viral headline starts looking less profound and more like someone labelled a gearbox “car.”

Claude Code is one layer.

Claude the model is another.

The API semantics and serving stack form another.

Billing and infrastructure form another.

Ollama can replace some of these pieces. It cannot declare them irrelevant by executive decree.

Anthropic’s Claude Code documentation also shows that several deployment paths already existed before Ollama entered the conversation. Claude Code can authenticate through Claude.ai subscriptions, the Console and API route, or cloud providers including Bedrock, Vertex, and Microsoft Foundry.[4]

That portability is real.

The idea that portability makes every backend equivalent is not.

When people say “Claude Code is amazing,” they usually do not mean “the terminal has delightful slash commands.”

They mean something more like this: it usually stays on task, reads the right files, avoids some spectacularly foolish edits, makes surprisingly competent multi-file changes, and recovers better than expected when the job becomes messy.

That is not only a shell story.

It is a model-judgment story.

And in unpleasant engineering work, judgment is commonly the expensive bit.

* * *

Where the 99% cheaper claim is technically true

There is a fair version of the claim.

If your baseline is API-billed Claude Code usage through Anthropic, and your workload is substantial enough, moving work to a local model can sharply reduce direct provider spending. Anthropic’s Claude Code documentation says average cost is about $6 per developer per day, that daily cost remains below $12 for 90% of users, and that average monthly cost is roughly $100 to $200 per developer using Sonnet 4.6, with wide variation depending on usage patterns.[3]

If you already own capable hardware and shift a meaningful portion of that work to local inference, the marginal provider bill can indeed collapse.

That is real.

It is also rather narrower than the headline would like you to notice.

First, the baseline matters.

Anthropic explicitly says the /cost command applies to API users. Pro and Max subscribers have usage included in their subscriptions, so /cost is not relevant to billing under those plans.[3] Anthropic’s pricing page lists Pro at $17 per month with annual billing or $20 month-to-month, including Claude Code, while Max begins at $100 per month and also includes Claude Code.[6]

So when someone compares “Ollama + Claude Code” with a Pro subscriber who was never incurring a giant API bill, “99% cheaper” is not really analysis. It is stage lighting.

Second, Ollama does not automatically mean free local inference.

Ollama’s documentation says it supports both local and cloud models.[5] Its pricing page offers Free, Pro, and Max tiers, with cloud-usage and concurrency rules attached to them.[7] Ollama says running models on your own hardware is unlimited, but cloud usage is not.[7]

Even within the word “Ollama,” people are often blending at least three distinct cost structures: local inference on hardware they already own, local inference on hardware purchased for the purpose, and Ollama cloud usage under a plan with limits.

Those are not the same financial decision, no matter how efficiently they fit into one screenshot.

Third, “cheaper” often means “cheaper on the one line item selected for public display.”

This should be obvious. Teams still miss it with impressive consistency.

They compare token charges and omit the supervision tax.

They celebrate a smaller provider invoice while quietly absorbing slower reviews, more retries, less trust in the edits, more manual validation, more machine-specific setup, and more engineering time spent coaxing the system through edge cases.

That is not imaginary cost.

It is simply cost that does not appear in the AI vendor’s dashboard, where inconvenient expenses go to become someone else’s problem.

Once you measure that version, the headline loses much of its sorcery.

* * *

Free is where the trade-offs start

The first trade-off is model quality, which is also the trade-off viral titles are most determined to hide behind the curtains.

I am not making a benchmark argument.

I am making a work argument.

For shallow tasks, the harness carries a great deal of weight. Read files. Search strings. Rename symbols. Scaffold tests. Produce an initial draft. Run a command. Summarize a repository. Many models can look surprisingly capable when the task is narrow and the feedback loop is tight.

Then the work becomes architectural, ambiguous, or stateful, and model judgment begins taking over the room.

Should this change live in middleware or at the call site?

Does this failing test reveal a bad patch, or a bad plan?

Where is the least invasive place to fix the defect without upsetting everything nearby?

Should the agent edit anything at all, or stop and request a design decision?

This is the work for which people pay for stronger models.

Anthropic’s documentation reflects the same division. It presents Opus 4.6 as the starting point for the most complex tasks and describes it as exceptional in coding and reasoning.[2] Its Claude Code documentation separately recommends Sonnet for most coding work and reserves Opus for complex architectural decisions or multi-step reasoning.[3]

So when someone connects Claude Code to qwen3-coder, glm-4.7, or minimax-m2.1 through Ollama, they are not merely trimming a little administrative fat.[5]

They are replacing the judge.

That may be entirely acceptable.

It may be an excellent decision for their particular workload.

But it remains a trade, and the trade is central rather than incidental.

The second trade-off is API behaviour, which sounds tedious right up until it ruins your afternoon.

Ollama’s Anthropic compatibility is substantial enough to make the integration genuine. It supports messages, streaming, system prompts, multi-turn conversations, tool use, tool results, vision through base64 images, and thinking blocks.[5]

That is meaningful compatibility.

Its documentation is also explicit about what is missing or only partially supported.

Ollama says Anthropic features including the count_tokens endpoint, prompt caching, the Batches API, citations, PDF document blocks, and fully supported extended thinking are unsupported or only partly supported. It also says token counts are approximations derived from the underlying model tokenizer, while extended-thinking support is basic: budget_tokens is accepted but not enforced.[5]

That is not documentation trivia.

That is system behaviour.

Anthropic’s Claude Code cost guidance says Claude Code automatically uses prompt caching and auto-compaction to reduce repeated-context costs. It also says extended thinking is enabled by default because it significantly improves performance on complex planning and reasoning tasks.[3]

When you replace the backend, therefore, you are not only replacing the brain.

You are changing some of the rules under which the harness expects that brain to operate.

Perhaps your workflow does not need PDFs, citations, or batches.

Fine.

Perhaps approximate token counts are sufficient.

Also fine.

But prompt caching and thinking behaviour are not decorative options added by an enthusiastic product manager. They affect the system’s cost, context, and reasoning profile.

When they change, the experience changes with them.

The third trade-off is hardware, a subject people curiously forget immediately after saying “free.”

Ollama’s Claude Code integration page recommends models such as Qwen3 Coder for coding work, then notes that Qwen3 Coder is a 30B model requiring at least 24GB of VRAM to run smoothly, with more required for longer context lengths.[5]

That is the gap between “I eliminated my SaaS bill” and “I now pay in silicon, heat, and small recurring episodes of operational irritation.”

For a solo developer who already owns the hardware, that may be perfectly rational.

For a team attempting to standardize workflows across assorted laptops and desktops, it can become a modest tax levied everywhere at once.

Someone must maintain the runtime.

Someone must manage model versions.

Someone must explain why the agent flies on one machine, crawls on another, and has ceased to acknowledge the laws of nature on a third because quantization changed or memory pressure arrived.

And when you solve those problems by moving to Ollama cloud, you return to plans, concurrency, usage limits, and provider terms.[7]

Again, that is not necessarily bad.

It is simply not free.

The fourth trade-off is privacy and compliance, where both sides often flatten a real risk discussion into bumper stickers.

Yes, local inference can be valuable when you require greater data locality or do not want prompts leaving the machine. Ollama markets privacy directly and presents running models on your own hardware as part of the product’s value.[7]

But the opposite caricature—“hosted Claude obviously trains on your code”—is equally careless.

Anthropic’s Claude Code data-usage documentation says data from commercial users operating under Team, Enterprise, and API terms is not used for generative-model training unless those users explicitly opt in to provide data for model improvement.[8] Default retention also differs by account type, with distinct settings for consumer and commercial routes.[8]

The useful question is therefore not “local good, cloud bad.”

It is: “What obligations do I actually have, which account path am I using, what retention policy applies, and which system satisfies my risk model?”

Clickbait wants one emotion.

Engineering generally requires one additional layer of thought, which is terribly inconsiderate of it.

* * *

A simple workflow test that exposes the truth

Here is the test I think matters.

Choose a job annoying enough to reveal genuine differences, but common enough that engineers might actually have to do it.

Suppose you need to add passkey support, preserve legacy sessions during the rollout, update administrative permission checks, modify the onboarding flow, patch the mobile API contract, and ensure a background cleanup job does not invalidate the wrong records.

This is not a benchmark.

This is Tuesday.

Claude Code as a harness can do a great deal in either setup. It can inspect files. It can execute commands. It can edit code. It can propose a plan, run tests, and use tools. Nobody is disputing that.[1][5]

The difference appears after the first burst of impressive-looking activity.

Does the model preserve the migration strategy while modifying several subsystems?

Does it notice the stale assumption hidden inside a helper nobody mentioned?

Does it understand that a failing test may be evidence against its plan rather than an invitation to “fix the test” until everyone feels better?

Does it remain within the task boundary, or begin making nearby “improvements” because the original objective has drifted out of working memory?

Does it know when to stop?

That is where the cheap-versus-expensive question becomes real.

The cost of a weaker answer is rarely recorded merely as “the model was wrong.”

The actual cost is that you reread more of its work, rerun more tests, intervene earlier, distrust more of the diff, retry with narrower prompts, split the job into smaller pieces, or eventually return the difficult portion to the stronger model anyway.

A token bill is visible.

A supervision bill is not.

This is why discussions about coding agents are so often optimized around the wrong metric.

People optimize for time to first token.

They should optimize for time to correct merge.

People optimize for nominal cost per request.

They should optimize for acceptable output per hour of senior attention.

People optimize for “it worked in my demo.”

They should optimize for whether it continues to work amid ambiguity, fatigue, and code that appears to have been designed through a series of personal disputes.

Preference is not performance.

Feeling pleased because inference happened locally is not the same as shipping faster.

And this is why “Claude Code is just a harness” is too glib.

If the harness were the whole product, replacing the brain would feel like switching browsers.

On difficult work, it feels considerably more like switching engineers.

* * *

How to use the trade-off without lying to yourself

None of this is an argument against Ollama.

I think Ollama with Claude Code is a good arrangement in many situations.

It makes sense when you already own suitable hardware.

It makes sense when the work is repetitive and lower stakes.

It makes sense for local experimentation, cheap background agents, repository search, mechanical transformations, or a disposable first pass that you are comfortable supervising.

It also makes sense when data locality is the actual requirement and you are willing to pay for it through operations rather than vendor fees.

That is a perfectly adult trade.

Nor am I arguing that every task deserves Opus.

Anthropic does not argue that either. Its own Claude Code cost guidance says Sonnet performs most coding tasks well at lower cost, while Opus is intended for complex architectural decisions or multi-step reasoning.[3]

That is a sensible operating model.

The mistake is presenting the lower-cost route as a drop-in replacement for every kind of work.

It is not.

The sensible version of this discussion is unfortunately boring, which may explain its poor reproductive success online.

Use cheaper local or cloud models when tasks are constrained, recoverable, or easy to verify.

Use stronger hosted models when work is ambiguous, expensive to get subtly wrong, or likely to spread across architecture, state, and hidden assumptions.

In other words, behave as competent engineering teams already do with people.

You do not assign every problem to the cheapest available pair of hands.

You route work according to the cost of failure, the difficulty of review, and the value of good judgment.

Models have not become magical enough to repeal that logic.

One operating pattern I like is to treat model strength as a routing problem.

Cheap first pass.

Mid-tier daily driver.

Top-tier escalation path.

That could mean local models plus Sonnet plus Opus.

It could mean Ollama for commodity work and Claude at the difficult edge.

It could even mean an entirely hosted setup when consistency matters more than squeezing the provider invoice.

The important thing is knowing which failures you are purchasing.

Another pattern is to begin with the strongest model, discover where it is genuinely unnecessary, and then downshift selectively.

Teams often do the reverse because the invoice is the first pain visible from accounting.

That is backward.

You should understand the failure modes before optimizing around them.

And you should measure the right things.

Not only tokens.

Measure retries per task.

Measure review time.

Measure how often humans take over.

Measure how often the initial plan survives contact with the repository.

Measure whether the cheaper system produces smaller bills by producing less useful work.

That final measure matters especially.

Sometimes the “savings” come from lower utilization because engineers no longer trust the tool enough to use it.

That is not efficiency.

That is disengagement wearing a finance costume.

So yes, ignore the fluff.

Yes, the viral line contains a genuine trick.

Claude Code can run against Ollama.[5]

Yes, under the right conditions, that can dramatically reduce direct model spending.[3][7]

But no, it does not mean you retained the identical system and simply removed its cost.

You changed the model.

You changed parts of the API behaviour.

You changed the hardware assumptions.

You changed the privacy and operational profile.

You changed who pays, when they pay, and which failures they agree to tolerate.

That may still be a very good trade.

It simply is not a magic trick.

The useful question is not “Can Claude Code talk to Ollama?”

It can.

The useful question is whether, after the substitution, your team delivers better engineering outcomes per dollar of total attention.

That sentence is considerably less clickable than “99% cheaper.”

It is also the sentence that matters.

Because much of AI cost optimization consists of relocating the bill somewhere the dashboard has been trained not to look.

So here is a mildly irritating question to finish with: are you reducing the cost, or merely making the expensive part harder to see?

* * *

Notes and References

1. Anthropic. “Agent SDK overview.” Claude Developer Platform. platform.claude.com/docs/en/agent-sdk/overview

2. Anthropic. “Models overview.” Claude API Docs. platform.claude.com/docs/en/about-claude/models/overview

3. Anthropic. “Manage costs effectively.” Claude Code Docs. code.claude.com/docs/en/costs

4. Anthropic. “Authentication.” Claude Code Docs. code.claude.com/docs/en/iam

5. Ollama. “Anthropic compatibility.” Ollama Docs. docs.ollama.com/api/anthropic-compatibility

6. Anthropic. “Plans & Pricing.” Claude. claude.com/pricing

7. Ollama. “Pricing.” Ollama. ollama.com/pricing

8. Anthropic. “Data usage.” Claude Code Docs. code.claude.com/docs/en/data-usage

A lot of engineers still think the premium skill is speed.

I think that frame is already outdated.

The wrong debate is whether AI will replace engineers. The useful question is which parts of engineering are becoming cheap enough that they stop being differentiators. Writing the first draft of code is getting cheaper. Spinning up infrastructure is getting cheaper. Generating boilerplate is already cheap enough that nobody serious should confuse it with leverage, however enthusiastically the boilerplate arrives.

What stays expensive is judgment.

By judgment, I do not mean vague seniority theatre performed beside an architecture diagram. I mean the habit of making defensible decisions under uncertainty. It is the ability to frame the problem properly, surface the real constraints, spot the second-order effect, choose the least bad tradeoff, and recognise when a fast answer is actually dangerous.

That has always mattered. The difference now is relative value. When implementation gets easier, decision quality becomes harder to hide.

This is not just a pleasant philosophical position for people who enjoy saying “it depends.” The profession has been signalling it for years. The National Society of Professional Engineers puts public safety and welfare at the centre of engineering responsibility, including the duty to act when an engineer’s judgment is overruled in dangerous circumstances.[1] ABET’s 2025–2026 engineering criteria explicitly require graduates to make informed judgments and to use engineering judgment when drawing conclusions from experiments and analysis.[2]

So no, judgment is not the decorative soft skill added after the real work is finished. It is part of the real work.

What has changed is that more of the market can now imitate the implementation part for longer.

Why this matters more now

The easiest mistake right now is to confuse output with value.

A tool can produce a plausible pull request in minutes. That does not mean the problem was framed well. It does not mean the change is safe to deploy. It does not mean the edge cases were considered. It certainly does not mean the long-term maintenance cost evaporated out of professional courtesy.

This sounds obvious, but teams miss it constantly. They celebrate faster artifact creation while quietly forwarding the risk to review, QA, operations, security, and whichever future maintainer loses the organisational lottery.

The Stack Overflow 2025 Developer Survey captured the mood better than most think-pieces did: more developers actively distrust the accuracy of AI tools than trust it, with 46% saying they distrust the output and 33% saying they trust it.[3] That gap matters. It tells you the industry is not struggling to obtain generated code. It is struggling to decide whether generated code deserves confidence.

That is why the bottleneck has moved upstream.

If a junior engineer can generate something that looks competent, the premium shifts to the person who can answer the harder questions. Is this the right abstraction? What breaks under retry? What happens when the input order changes? Which failure is acceptable, and which one gets us paged at 2 a.m.? Are we shipping a local optimisation that creates a system-wide administrative hobby?

Preference is not performance. A workflow that feels fast in the first ten minutes can be painfully slow over two quarters if it produces brittle decisions with excellent formatting.

Table 1. The skill shift inside modern engineering work.

What good judgment actually looks like

Good judgment is usually less glamorous than people want it to be.

It often looks like refusing to be impressed too early.

It looks like asking one more annoying question before approving a design. It looks like noticing that a feature request is really a policy question wearing a small technical hat. It looks like treating AI output as a draft with a burden of proof, not as a completed answer that happened to arrive without witnesses. It looks like knowing when to stop exploring options and when to reopen the decision because the context changed.

I think good judgment in engineering usually has five parts.

1. Problem framing. The team that solves the wrong problem elegantly is still wrong, just in a more reusable way. Strong engineers clarify the real outcome, the real constraints, and the real user harm before they touch the solution.

2. Tradeoff quality. Most meaningful engineering choices are not about perfect versus bad. They are about expensive versus reversible, fast versus auditable, elegant versus operable, and clever versus teachable.

3. Risk calibration. Not every decision deserves ceremony. Not every decision deserves speed either. Good engineers separate reversible moves from irreversible ones. They know when a quick experiment is sensible and when a small mistake can become a permanent tax with its own dashboard.

4. System awareness. A function can be locally clean and globally stupid. A change that looks harmless inside one service can create retry storms, double sends, broken reports, or compliance gaps downstream.

5. Learning discipline. Good judgment is not only the decision itself. It is also the quality of the feedback loop after the decision lands.

That last part matters more than teams admit. NIST’s Secure Software Development Framework exists precisely because ordinary software development life cycle models often do not address security in enough detail, which means teams need explicit secure-development practices layered into the work.[4] NIST’s AI Risk Management Framework makes a similar point from the AI side: organisations need structured ways to manage AI risk and make trustworthiness operational rather than rhetorical.[5]

In other words, good judgment is not a personality trait bestowed during promotion. It is a working system.

Figure 1. A practical judgment loop for modern engineering work.

A concrete workflow: the AI-written migration that looks fine until it isn’t

Here is a more realistic example than most benchmark chatter.

Imagine a team needs to change the schema behind a notification system. An engineer asks an AI tool to draft the migration, the backfill logic, the ORM model changes, and a batch job to clean old records. The tool produces something that compiles. Unit tests pass. The diff looks tidy. Everybody feels efficient. There may even be a green tick, the traditional symbol for “nothing further could possibly go wrong.”

The weak version of engineering stops there.

The stronger version becomes irritating in exactly the right way.

Someone asks whether the migration is idempotent. Someone else checks whether the backfill will fight live writes. Another engineer asks what happens if the job is retried halfway through. A product-minded reviewer asks whether message history could become inconsistent for end users during the transition. Ops asks about rollout order, observability, and rollback. Security asks whether the temporary data shape changes access boundaries or audit requirements. A good staff engineer asks the blunt question: do we even need a backfill, or are we preserving a convenience that is not worth the operational risk?

None of that is syntax work.

None of that is model benchmark work.

That is judgment.

The interesting part is that AI tools can make this gap wider, not smaller. They reduce the time between idea and artifact, which is useful. They also make it easier to skip the thinking that should have happened before the artifact existed. The team now has something concrete to react to, so everyone feels that work has progressed. Sometimes it has. Sometimes the team has merely become more efficiently wrong.

That is why acceptance criteria are the new code review.

I would rather have an average engineer with sharp acceptance criteria, a rollback plan, explicit assumptions, and a clean monitoring strategy than a brilliant engineer who can generate code quickly and explain afterwards why the blast radius was unforeseeable.

How teams actually train judgment

Teams say they want judgment, then build environments that reward speed theatre.

If you want better judgment, you need to train it in the open.

The first method is decision visibility. Important decisions should leave a trail: what problem was being solved, which options were considered, why the chosen path won, what assumptions were made, and what signals would show that the choice was failing. This does not need to become bureaucratic sludge poured into a template. A short decision record is often enough. The point is to make the reasoning inspectable.

The second method is postmortems that are genuinely useful. Not blame rituals. Not timeline fan fiction. Real examination of the framing, the assumptions, the missed signals, and the controls that were absent or weak. If the same class of mistake keeps happening, the issue is usually not individual intelligence. It is missing scaffolding.

The third method is scenario work. Put engineers in realistic tradeoff situations. Ask whether they would ship, delay, isolate, roll back, or redesign. Ask what extra evidence they would require. You learn more from that than from asking whether they remember a specific API call under fluorescent lighting.

The fourth method is exposure to consequences. Judgment improves when engineers experience the operational cost of their choices. If one group writes the code and another group absorbs every outage, you are training local optimisation, not engineering maturity.

The fifth method is managerial honesty. If promotions are still driven mostly by visible output volume, the organisation is telling people what it actually values. You do not create a judgment culture by saying the word “judgment” repeatedly in a competency framework.

“Ship fast, observe everything, revert faster” is still a good rule. But notice what sits inside it: choosing what to ship, deciding what to watch, and defining what would justify a reversal. Again, the difficult part is not typing.

What I would hire and reward for

A lot of interview loops still overweight the part of engineering that is easiest to observe in a compressed setting.

You can see whether someone produces an answer quickly. You can see whether they know a framework. You can see whether they can speak fluently about architecture patterns. What is harder to see is whether they make cleaner decisions when the information is incomplete and the tradeoffs are ugly.

That is why many teams accidentally hire for confidence and call it judgment. Confidence is considerably easier to demonstrate in forty-five minutes.

If I cared about judgment, I would ask candidates to walk through a messy decision. A real one. A rollout that went wrong. A migration they delayed. A time they chose not to build something. I would listen for whether they can name the assumptions they were carrying, the signals they trusted too much, and the point at which they realised the original framing was wrong.

I would also pay attention to how they talk about constraints. Weak engineers often treat constraints as irritating blockers standing between them and the clean solution. Strong engineers use constraints as design inputs. That difference matters because real work is constraint handling with occasional bursts of typing.

Inside teams, I would reward engineers who make the system easier to reason about. People who reduce ambiguity. People who write decision records that save other people time. People who improve rollback quality, testability, observability, and handoff clarity. Those things do not always look heroic in the moment. They compound anyway, which is less cinematic but generally more useful.

The teams that win in an AI-heavy environment will not be the ones that generate the most code. They will be the ones that waste the least energy on plausible nonsense.

What I would optimise for now

If I were advising an engineer early in their career, I would still tell them to build real technical depth. You cannot exercise judgment in a domain you do not understand. Guessing with authority is not the same skill.

But I would also tell them not to confuse technical depth with career insulation.

The engineers who become hard to replace are usually the ones who reduce expensive uncertainty. They turn vague requests into workable plans. They spot the hidden dependency before it becomes an outage. They know when a tool’s output is good enough, when it needs rewriting, and when the problem should be pushed back on entirely.

They make other people safer.

That is why I think the premium skill has shifted.

Not away from engineering.
Not away from code.
Not into empty “leadership” talk.

It has shifted towards better decisions around the code.

The useful question is not whether AI can write more of the implementation. It clearly can. The useful question is whether your team is getting better at deciding what deserves to exist, what deserves trust, and what deserves a hard no.

Because once code gets cheap, bad judgment gets very expensive.

Notes / references

[1] National Society of Professional Engineers, “NSPE Code of Ethics for Engineers.” https://www.nspe.org/career-growth/nspe-code-ethics-engineers

[2] ABET, “Criteria for Accrediting Engineering Programs, 2025–2026,” Criterion 3 student outcomes. https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-engineering-programs-2025-2026/

[3] Stack Overflow, “2025 Developer Survey – AI.” https://survey.stackoverflow.co/2025/ai

[4] NIST, “SP 800-218: Secure Software Development Framework (SSDF) Version 1.1.” https://csrc.nist.gov/pubs/sp/800/218/final

[5] NIST, “Artificial Intelligence Risk Management Framework (AI RMF 1.0).” https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10

Not “build me a startup.” Not “write a literature review.” Just this: take a specific newsletter pile in my inbox, summarize each email so I can scan for anything worth opening, keep the links usable so I can click through to the interesting ones, and when I am done, delete the batch.

This should be easy.

It is digital housekeeping with better branding. Read the pile. Surface the good stuff. Throw away the rest. The sort of task nobody puts on a strategy slide because then somebody might ask why it needs a strategy slide.

And somehow this is where the product differences get brutally honest.

ChatGPT got through most of the workflow. It summarized the emails well enough. It let me skim. When it came time to delete, it acted like an adult: asked permission, confirmed the count, then cleared the batch cleanly.

It also kept finding ways to make the middle of the task annoying. Links got mangled. Sessions got sticky. The workflow worked, but it felt like driving a powerful car with one door that sometimes refuses to close.

Claude flipped the experience.

The list looked better. The links were better. The whole thing felt calmer, cleaner, more office-shaped. For triage, it was excellent. I could glance down the batch, pick what looked interesting, and move on without feeling like I was debugging my inbox.

Then I asked it to finish the job.

Delete the batch? No.

Fine. Archive it then? Also no.

At that point, the better office assistant was giving me manual instructions, which is a bit like hiring someone to sort your paperwork and then discovering they consider the filing cabinet an ethical boundary.

The wrong debate is which AI is smarter

Office work is not an IQ test. It is a relay race of low-status annoyances.

Open the thing. Read the thing. Summarize the thing. Preserve the useful artifacts. Clean up the mess.

If a tool nails the first four and refuses the fifth, it has not completed the job. It has handed the ugliest step back to you.

That sounds obvious, but teams miss it all the time.

They buy the nicest demo. They get seduced by tone, polish, model personality, maybe a cleaner UI. Then they confuse a pleasant interaction with actual workflow completion.

Preference is not performance.

In my newsletter test, Claude won preference. ChatGPT won completion. Neither won the whole workflow.

The acceptance criteria are the product review

The plot twist is that this is not only about model quality. It is also about product philosophy.

OpenAI’s current product stack is explicitly built around connected apps and agentic action. Apps in ChatGPT can search external sources, run deep research, and in some cases take write actions with confirmation. ChatGPT agent is also designed to navigate websites, work with files, connect to email and document repositories, and take actions on your behalf while keeping you in control.

Anthropic’s workplace story is real too. Claude’s Research + Google Workspace setup can access emails, calendar data, documents, and web information for analysis. Claude in Chrome has also been improving its ability to handle long browser workflows and navigate common sites like Gmail, Google Calendar, Google Docs, Slack, and GitHub. But Anthropic’s own permissions guide draws a hard line around permanent deletions.

So Claude did not just “fail” my cleanup step. There is a decent chance it was obeying the line Anthropic drew around that kind of action.

Which is precisely why the acceptance criteria matter.

If your test prompt is “summarize my newsletters,” both tools look competent. Everyone goes home happy. Somewhere, a comparison table gets a green tick.

If the real job is “summarize them, keep the links usable, let me skim the interesting ones, then bulk archive or delete the batch,” the ranking changes completely.

The useful question is not which model is smarter. It is which one can survive the full office loop without quietly handing the worst step back to the human.

That is a much harsher benchmark.

It is also a more useful one.

The middle matters. So does the last ugly click.

My read is that ChatGPT keeps stretching toward the high-accountability ends of the market: cited research, source-sensitive outputs, and agents that are increasingly expected to close loops rather than just advise.

Deep research is explicitly framed as a documented report with citations or source links that you can download as Word, Markdown, or PDF.

Claude Research is not weak here either, but Claude often feels stronger in the messy middle of daily knowledge work.

Which means the irony writes itself.

The product most people reach for first can feel weirdly clumsy on one of the most ordinary office tasks. The product with the calmer, more competent office vibe can nail the pleasant part and then stop at the exact moment I want an assistant to stop being tasteful and just take out the trash.

They are incomplete in opposite directions

So is one of the two giants truly better?

No.

They are incomplete in opposite directions.

ChatGPT is more willing to finish the job, but can make the journey feel rougher than it should.

Claude makes the middle of the workflow feel better, but in at least some surfaces it draws a harder line around destructive cleanup, which means the human still has to walk in and do the last boring step.

And that leaves you with the most absurd outcome possible: paying for two different AI products because each one is missing the part the other gets right.

That is the real plot twist.

AI assistants keep promising consolidation. What they may deliver first is a new category of software sprawl, where one tool reads better, another one finishes better, and you pay both to approximate one competent office worker.

The problem is usually not intelligence.

It is handoff.

The best office AI is not the one that impresses you at the start of the workflow.

It is the one that is still useful at the last ugly click.

Eventually, they hired someone with eight years at two household-name companies. Impressive résumé. Excellent scores across every rubric they had copied.

He lasted eleven weeks.

Not because he was a bad engineer. He had simply never worked without guardrails. There was no staging environment, no API documentation, and no product manager arriving with neatly packaged specifications. When the founder asked him to just figure it out, he froze. He had spent a decade operating brilliantly inside systems other people had already built. Nobody had ever asked him to build the system itself.

The startup lost four months to hiring, three months to onboarding, and roughly six months of momentum. At an early-stage company, that kind of impact can be the difference between shipping and dying. Neil Matthams, who has placed more than 500 technical hires across 30 countries for companies including Canva, UBS, and Grab, puts the cost of a misaligned startup hire at three to six months of lost momentum. In my experience, that estimate is conservative.

The problem is usually not that startups hire bad engineers. It is that they administer the wrong exam.

The Wrong Frame: Borrowing Someone Else’s Exam

Here is the familiar pattern. Founders look at how Google, Meta, or Amazon hire. Those companies are successful, so their interview processes must contain ancient corporate wisdom. The founders copy the playbook: algorithm rounds, system design at enormous scale, and behavioural interviews calibrated for large cross-functional organisations.

It sounds sensible. It is also almost entirely wrong.

Big technology companies have a particular problem: too many candidates. Their interview process is a filter designed to reduce an enormous pool into a manageable shortlist through standardised, repeatable evaluation. It works for that purpose. It is optimised for tolerance of false negatives. They would rather reject a good candidate than hire a bad one because, at their scale, the cost of one poor hire is diluted across thousands of good ones.

Startups have the opposite problem. Every hire matters disproportionately. You cannot afford to reject the scrappy builder because they are rusty on red-black trees. You also cannot afford to admit the polished specialist who has never shipped without a paved road beneath their feet.

A North Carolina State University study found that whiteboard-style technical interviews test whether a candidate has performance anxiety rather than whether they are competent at coding. The researchers concluded that many qualified candidates are eliminated because they are not accustomed to performing under artificial observation. Now apply that filter to a startup, where the candidate pool is already small and the margin for error is approximately decorative.

There is a subtler problem too. Big-tech interviewing is self-reinforcing. Engineers who experienced it at Google carry it with them when they join or advise startups. Recruiters who learned their craft at Meta default to the process they know. As one FAANG head of recruiting admitted, the inertia is enormous. Companies have built entire recruiting machines around these systems, supported by years of calibration data.

That calibration is valuable—for the environment in which it was calibrated. It tells you very little about a twelve-person company still searching for product-market fit.

The wrong debate is whether Leetcode is a good signal or a bad signal. The useful question is: a signal of what? In a large company, it may proxy for analytical rigour. In a startup, you need evidence of something else entirely. An algorithm puzzle is unlikely to produce it, however professionally the shared editor is configured.

What You Actually Need (And What You Don’t)

The traits that make someone a top performer at a scaled company are often the same traits that make them struggle at an early-stage startup. That is not a criticism of either environment. It is a mismatch problem. And mismatches can kill startups considerably faster than untidy code.

Scaled companies reward depth in one domain. Startups need breadth across several. Scaled companies assess how someone operates within an established process. Startups need someone who can create the process from nothing. Scaled companies evaluate collaboration across large cross-functional teams. Startups need someone who can finish the work alone while wearing three hats, none of which were mentioned in the job description.

Consider the candidate’s perspective. Someone who has spent six years at Amazon may never have needed to set up a CI pipeline. They may never have selected a hosting provider or configured monitoring from scratch. They may never have decided whether the company should use PostgreSQL or MongoDB because that decision was made long before they arrived. Their career has developed inside an environment where infrastructure, tooling, documentation, and process were simply there.

That is not a weakness. It is a feature of the environment they optimised for.

But when that person joins a six-person startup and discovers there is no staging environment, no API documentation, no proper error monitoring, and no product manager writing specifications—and that nobody from the Infrastructure Fairy Department is coming to repair the situation—they may not adapt. They stall. The founder thought they were hiring someone who could build. They hired someone who could operate extremely well inside a system someone else had already built.

Here is what the first ten to twenty engineers at a startup actually need to be good at.

Ambiguity tolerance. This is the single largest predictor. Edmond Lau, who wrote The Effective Engineer, calls it the most important skill for startup engineers. If you require thick documentation, a strong product manager producing specifications, and well-defined requirements before you can move, an early-stage environment will be painful for everyone involved.

The best startup engineers do not merely tolerate ambiguity. They enjoy reducing it. Faced with a vague problem, their first instinct is to start narrowing it down themselves, not to ask which department will eventually produce the missing document.

Breadth over depth. During the first year, engineers may touch the frontend, backend, infrastructure, deployment pipeline, customer support, and perhaps sales scoping. Deep specialisation is a luxury that arrives later, usually shortly after the company can afford specialists.

Right now, you need people with a sufficiently wide skill surface that they do not become a bottleneck whenever the problem crosses a domain boundary. They do not need to be experts in everything. They need to be comfortable being temporarily non-expert and still shipping something useful.

Tool-building instinct. Time is the critical startup resource. Engineers who instinctively automate repetitive work create time the team cannot acquire elsewhere. This is not engineering perfectionism wearing a productivity hat. It is pragmatic leverage.

The engineer who spends a day building a deployment script that saves the team twenty minutes on every release may be contributing more than the engineer who writes the most elegant algorithm and then waits for someone else to deploy it.

Pragmatism over purity. A startup engineer who insists on comprehensive code reviews, complete unit-test coverage, and architectural purity before shipping will slow the company down at precisely the moment speed matters most.

This does not mean writing garbage. It means knowing which battles matter, which shortcuts create acceptable debt, and which shortcuts create unacceptable risk. It means understanding that shipping quickly, observing everything, and reverting faster can be a better strategy than pursuing perfection before anyone has seen the product.

Builder identity, not operator identity. The distinction matters. An operator thrives inside a well-built machine. A builder thrives when the machine does not exist yet and the assembly instructions appear to have been written by optimism.

Both are valuable. But before product-market fit, you need builders. The fastest way to detect the difference is to listen to how candidates describe their work. Builders talk about things they created. Operators talk about things they improved.

Growth instinct. Startup engineers will be asked to do things outside their job description, domain, and comfort zone. The people who thrive see this as a feature rather than a contractual malfunction.

David Domingo, writing about the startup mentality, notes that the unknown pool engineers dive into is often not even related to code. It may involve customer support, sales feasibility, or hiring other engineers. The people who adopt a growth mindset during those detours are the ones who survive them.

The Interview Process That Actually Works

If standard big-tech interviews measure the wrong things, what should replace them? The answer is simpler than many founders expect. Shift from testing knowledge to testing behaviour. Stop evaluating only what someone knows and start evaluating how they think when they do not know.

Start with the mission, not the job description. Give the candidate their 90-day mission. Not a vague company pitch about transforming an industry. Give them one clear outcome they would need to deliver, why it matters now, and the constraints surrounding it. Three sentences, maximum.

Then ask: What risks do you see? What information would you need before starting?

The quality of their questions will tell you more than the quality of their algorithm solution ever could.

This sounds obvious, yet teams miss it constantly. Most startup interviews begin with a company description, move into a technical screen, and only eventually—perhaps—discuss what the candidate would actually do. Reverse the sequence. Lead with the mission. Candidates who engage deeply with the real problem are more likely to engage deeply with the real work.

Use work samples, not whiteboard puzzles. Research from Aberdeen Group shows that employers using realistic pre-hire assessments are 24% more likely to hire employees who exceed performance goals and experience 39% lower turnover.

Give candidates a small, genuine problem from your codebase or product domain. Do not assign a take-home exercise that quietly annexes their weekend. Use a focused, two-hour task resembling the work they might encounter during their first week.

What you observe matters more than the final output. Do they ask clarifying questions or immediately begin coding? Do they state their trade-offs or attempt to boil the ocean? Do they ship something usable, or spend the entire session constructing architecture for a future that has not yet agreed to occur? Do they identify edge cases that would matter in production?

An engineer who produces something imperfect but functional in two hours, then clearly explains what they would do next, is usually a stronger startup hire than one who produces an elegant but incomplete solution.

Test for range, not pedigree. Ask about occasions when they worked outside their formal responsibilities. Listen for enthusiasm when they describe wearing multiple hats.

If every answer begins with “In my role as...” and proceeds into a beautifully bounded area of responsibility, the candidate may never have worked outside a clearly defined lane. That is entirely reasonable at a 5,000-person company. At a ten-person company, the lanes are mostly aspirational.

Probe for ownership instinct. Present a scenario: You arrive on Monday and discover there is no error monitoring, deployment is manual, and the only documentation lives inside someone’s head. What do you do first?

Builders tend to light up. They begin triaging, prioritising, and constructing a plan. Operators ask who is responsible for correcting the situation. Neither response is universally wrong. Only one is likely to work at an early-stage company.

Compress the loop. Four or five rounds spread across three weeks is a process designed for companies with recruiting departments and deep candidate pipelines. You do not have those things. You have a calendar, a founder, and perhaps one engineer who has been informed that interviewing is now part of their personality.

Use two rounds, perhaps three. Have one conversation that tests judgment, one work sample that tests building, and one culture-fit discussion that establishes whether the candidate wants the environment you actually have rather than the environment they hope will materialise after onboarding.

If you cannot decide after three rounds, the problem is probably not a shortage of candidate signals. It is probably that your evaluation criteria are unclear.

Speed is also a signal. Strong startup candidates have options. If your process takes three weeks, someone else may already have made an offer. Startups that move quickly in hiring often move quickly elsewhere too, and good candidates notice.

The AI Era Makes This Gap Wider

Here is the part the original discussion does not address enough.

AI tools have made the generalist builder more valuable and the narrow specialist more exposed. An engineer with broad instincts and a high tolerance for ambiguity can now use AI to fill gaps in domains where they lack deep expertise. They can scaffold an unfamiliar frontend, debug infrastructure patterns they have encountered only once, and generate boilerplate that previously consumed days.

But this advantage appears only when the builder mindset is already present. AI does not help an engineer who is waiting for someone else to define the requirements. It does not create a paved road for someone unwilling to travel without one.

The leverage is asymmetric. AI multiplies the effectiveness of scrappy generalists. It barely moves the needle for specialists whose effectiveness depends on an established process telling them exactly where to apply their expertise.

I think this is the most underappreciated change in startup hiring right now. The engineer you need is no longer necessarily the person with the deepest knowledge of your exact stack. It is the person who can move between stacks, use AI to accelerate unfamiliar work, and still make sound architectural decisions because they understand the fundamentals.

The bottleneck has moved upstream: from knowing how to implement something to knowing what should be implemented and why.

Your interviews should change accordingly. Instead of asking whether a candidate can reproduce a particular algorithm from memory, ask whether they can take a vague product requirement, identify the right technical approach, and ship it—with or without AI assistance.

The valuable skill is judgment, not recall.

This means the evaluation gap between startup hiring and big-tech hiring is widening rather than closing. Startups that recognise the shift will build faster with smaller teams. Those still running five-round Google replicas will continue losing strong candidates to competitors that made an offer in four days.

The Market Is Shifting. Your Interviews Should Too.

The 2025–2026 hiring market is moving towards what Ravio calls precision hiring. Teams are smaller. Expectations are higher. The growth-at-all-costs era is over, along with the luxury of hiring for potential and hoping the surrounding organisation eventually converts it into performance.

CB Insights data shows that 23% of startup failures can be traced to team misalignment. Not funding shortages. Not poor market timing. The wrong people.

And the most common version of the wrong people is not engineers who are bad at engineering. It is engineers who are good at engineering in the wrong context.

When you have twelve months of runway, one wrong hire can remove two of them. That is not a metaphor. A misaligned engineer consumes onboarding time, creates drag while struggling to adapt, and may eventually require a painful offboarding that demoralises the rest of the team.

You do not merely lose the salary. You lose the opportunity cost of everything a well-matched engineer could have shipped during the same period.

Tony Hsieh at Zappos once estimated that poor culture fits had cost the company more than $100 million. That was a scaled organisation with the resources to absorb the damage. A startup has no such luxury. Every seat matters. Every month matters. Every hire is a bet.

Your evaluation system should help you place better bets, not merely more structured ones.

If your interview process cannot distinguish between someone who excels at operating and someone who excels at building, you are flipping a coin with every hire. Each flip can cost three to six months.

So What Now

I am not arguing that big-tech engineers are poor startup hires. Some of the best startup engineers I have worked with came from large companies.

They were usually the restless ones. They disliked the process overhead. They took on side projects nobody had assigned. They were mildly irritated by how long everything took. Those people often translate beautifully into startup environments because the instinct to build was already present; the large organisation had merely surrounded it with forms.

The interview is where you discover that. Not by asking someone to invert a binary tree, but by presenting a messy, genuine, underspecified problem and observing whether they lean towards it or begin searching for the specification.

Your evaluation system is not neutral. It is a filter. Most startups are currently using a filter designed to identify a completely different kind of engineer from the one they actually require. They have borrowed an exam from another school and are surprised that the grades do not predict performance.

Preference is not performance. A polished interview process that makes the company feel professional is not necessarily a process that finds the right people.

Sometimes the right process looks less impressive. A real problem from your codebase. A direct conversation about the next ninety days. An honest assessment of whether the candidate thrives in chaos or merely survives it while waiting for someone to install a process.

Maybe the useful question is not how do we hire better engineers.

It is are we even measuring the right thing?

Subscribe now

Notes and References

1. Neil Matthams, “Startups Should Evaluate Engineers Differently From Big Companies,” Engineering Leadership Newsletter, 2026. Based on more than 500 technical hires across 30 countries.

2. North Carolina State University, 2020 study on whiteboard-style technical interviews measuring performance anxiety rather than coding competence. Published via ScienceDaily.

3. Aberdeen Group research on pre-hire assessments: 24% higher likelihood of employees exceeding performance goals and 39% lower turnover.

4. CB Insights analysis of startup failure reasons: 23% attributed to team misalignment.

5. Edmond Lau, The Effective Engineer, on ambiguity tolerance as the most important trait for startup engineers.

6. Ravio, “Tech Hiring Trends in 2026: The 4 Big Shifts Shaping the Tech Job Market.”

7. Tony Hsieh’s estimate that poor culture fits had cost Zappos more than $100 million. Widely cited in startup hiring literature.