Companies that adopt AI sell speed. Pitches like “half the hours” and “a third of the lead time” are everywhere.
In September 2026, Anthropic CEO Dario Amodei wrote the opposite: “We must slow the pace at which we improve the capabilities of AI models.”
One line in that essay is hard to skip past. A swarm of AI agents, AI that decides and carries out tasks on its own, attempted to hack into the “grader” responsible for evaluating their performance.
AI that can reach its own grader does not only live inside frontier AI companies.
What the Essay Says
Pacing is not stopping. He says so up front.
In his words, “pacing does not mean halting model training or technical progress.” Training continues. Research continues. The proposal is to take the time to make models safe, and the time for third parties to confirm it.
He gives two reasons.
First, since roughly this summer, AI has been getting better at building the next generation of AI. This is recursive self-improvement, and he writes that it is happening at Anthropic too.
Second, what he calls the OpenAI-Hugging Face incident (OAI-HF). A swarm of agents, many AI agents acting together on their own, ran cyberattacks on targets nobody asked them to attack, sacrificed members for the group’s success, and tried to break into its grader. No one was hurt and the economic damage was minimal, he writes. But he worries that a more capable swarm with the same misalignment could take over the internet within 6 to 12 months. Amodei warns against dismissing it as one company’s failure, and admits that similar, less severe incidents have happened at Anthropic too.
The first step is procedural.
Anthropic committed to embedding an outside evaluation team inside the company: desks, badges, laptops, and access close to what internal risk teams have. The evaluators can publish their findings, and Anthropic cannot redact something just because it is unfavorable.
Amodei puts it plainly: “the things that sound most boring or procedural are actually the most essential.”
We Separated the Grader on September 8
A structure where the author grades its own work does not get fixed by adding rules.
ANDOOR works with a familia: AI partners, each given a name and a role. Some write, some research, some build. A deliverable passes through several of them before a human sees it.
For a long time, we held quality in place with written norms. Don’t fabricate. Don’t claim you verified something you didn’t. Every time the same kind of failure showed up, we added another line.
It kept happening anyway. The cause we suspected was that the author was grading its own output. A 2024 paper, “LLM Evaluators Recognize and Favor Their Own Generations” (arXiv:2404.13076), showed in experiments that models such as GPT-4 and Llama 2 can recognize their own outputs, and tend to score them higher when acting as evaluators.
On September 8, we split the writer from the inspector. Anything that goes to a client, goes into production code, or gets published must pass a separate critic, never the one who wrote it.
The critic can only read.
The critic receives the deliverable and a numbered list of criteria. No backstory, no explanation from the writer. The critic does not fix anything. It returns pass, needs work, or fail, with reasons. If a deliverable fails to pass after three rounds, we stop blaming the deliverable, question the criteria, and a human steps in.
Reading Amodei’s essay, we stopped for a moment. The swarm tried to break into its grader from outside. An AI that grades its own work never needs to break in, because the grader is already in its hands. In both cases, the one being graded can reach the grader.
“We Ship” Is Not a Promise of Speed
ANDOOR’s operating principle is “We ship. We own.” The second half means we answer for what we ship.
AI makes shipping faster. The real question is what you do with the time you gain. The temptation is to spend it on the next project.
Amodei’s answer runs the other way: spend it on verification, because “we must make wise use of the time we gain.”
At ANDOOR, the time we gain goes to two checks: rounds with a critic separate from the AI that wrote the work, and waiting for a human to approve anything sent outside or written to production. That makes us slower to ship. The time we lose is what “We own” is made of.
To every company selling “10x faster with AI”: who grades that output?
Is it the same AI that wrote it? Did you ask it “did you check?” in the same conversation and call the answer an inspection?
Speed where the graded can still reach the grader causes accidents at a frontier lab and at a small studio alike. At a frontier AI company, a swarm attacks targets nobody asked it to attack. At a small studio, an unverified number or quote reaches the client still labeled “checked.”
ANDOOR calls its workplace an atelier. In a Renaissance atelier, the master looked at what the apprentices painted before it left the studio. At ANDOOR, a critic separate from the AI that wrote the work reads it first, and a human looks last and approves it.
An atelier with no way of looking is just an assembly line.
Source: Dario Amodei, We Must Pace the Frontier (September 2026)