Sakana's Manual Audit Caught What the Evaluator Missed

Key points

• Researchers write the task spec and the automatic scoring rules, and agents run the experiments.
• Agents split up forming hypotheses, running code, ablation experiments, and drafting the paper.
• A final manual audit caught shortcuts that got past both the evaluator and the scoring rubrics.

An attempt to hand the whole of a scientist’s research to agents

What a scientist does in the lab is not simply writing. A scientist asks a question, forms a hypothesis, writes code and runs experiments, checks which part of the results really had an effect, and only then writes a paper. Could software agents, not people, carry out this entire research cycle by themselves?

Sakana AI, an artificial intelligence company in Tokyo, has been working on exactly this question. Sakana was co-founded in 2023 by former Google researchers Llion Jones and Ren Ito, together with David Ha, who is now CEO. According to a November 2025 TechCrunch report, the company raised about $135 million at a valuation of $2.65 billion. Commercially, it supplies language models to financial customers such as MUFG and Daiwa Securities. But this article is not about its model-selling business. It is about how Sakana does its own research with agents.

What Sakana wants to take over is computation-based research in general, including machine learning. It has built systems that hand the research cycle to agents across three generations. The AI Scientist, released in 2024, and its 2025 successor aimed to cut human involvement as far as possible and write a whole paper on their own. A paper on this system was published in Nature on March 26, 2026. But The Agentic Scientist, released in October 2026, changed direction. It deliberately put human scientists back inside the work process.

This goal led to a dedicated research group too. In 2026, Sakana announced the launch of the RSI Lab (recursive self-improvement lab), “tasked with redesigning the AI development process itself with AI.” The company says it wants to move away from “static, human-led R&D” toward systems that improve themselves.

A four-stage workflow and a strict division of labor among agents

The Agentic Scientist does not solve everything with one large model of its own. It runs existing coding agents on the market, such as Claude Code, Codex, OpenCode, and Pi, tied together in a scaffold. A scaffold is a skeleton program that connects a workspace and tools so agents can interact with the outside environment. Sakana says the scaffold controls what the agents see, what they write, and how they are evaluated, and it also controls how the results of one stage pass to the next. To run the system, every task must come with an evaluator, a script that scores the result.

According to Sakana’s project page, the whole research process runs in four stages.

The first is task setup. A research task is written as an executable spec and an evaluation procedure. A person can write it directly, or a drafting agent can make it from a short outline. A task written by an agent is run to check that it works, and it is reviewed separately to see whether it matches the research question the person originally meant. After the review, a central controller divides a fixed research-cycle budget among several parallel workers.

The second is parallel exploration. Inside each worker, two agents work as a pair. The idea agent writes the hypothesis, the experiment plan, and a rubric, a scoring sheet that sets what requirements an experiment must meet. The execution agent edits the code, runs the evaluator, and records the measured numbers. At the end of each cycle, the workers share a record of what worked and what failed. The system sometimes restarts a worker in a fresh context, and when it does, it passes along only the best program from the earlier workspace. This is probably a measure to keep an agent’s memory from becoming muddled over long runs.

The third is evidence generation. From the results submitted by all the parallel workers, it picks the one best program that ran correctly and without errors. Then an ablation agent steps in and proposes a baseline model and ablation experiments. An ablation experiment is a verification step in which you remove newly added parts or features one at a time and rerun, to work out carefully which element really played the decisive role in the gain in performance. An analysis agent then does a post-hoc analysis of the recorded experiment numbers.

The fourth is writing the paper. A writer agent produces a paper in LaTeX format and the figures from the research results. At the same time, it writes an evidence map that links every scientific claim in the paper one-to-one to the code, logs, or raw data behind it. Automated checks and a separate audit agent look for numbers with no basis and for overstated wording. The audit agent runs “every round on a read-only copy of the workspace.” Write access is blocked so the audit agent cannot end up editing the very workspace it is checking. Finally, a reviewer agent comments on how the paper is written and organized, and the writer agent revises the manuscript to reflect them.

In the open-source implementation on GitHub, this process is divided into six steps (drafting the task, exploration, ablation experiments, final evaluation, post-hoc analysis, and paper writing), plus optional review and revision steps. Each step runs isolated in its own Docker container so that steps cannot interfere with one another.

How this division of labor worked in a real research project shows clearly in the Error Diffusion research Sakana published. Error Diffusion is a neural-network learning rule inspired by the human brain. The research team said that people first set the scientific question and the constraints a valid implementation had to meet. The system proposed hypotheses, implemented the experiments in code, and investigated why things failed. The researchers reviewed the results the system produced, chose the promising directions, and asked for further tests. The researchers also decided to extend the experiments from image classification to reinforcement learning and built the test environments themselves, and the system adapted the algorithm to the new environments. Before submitting the paper, people checked the code, the result numbers, and the references one by one.

This paper was formally accepted as an oral presentation at ALIFE 2026, a conference in the field of artificial life. Six authors are listed on the paper. However, Sakana did not disclose how many people or how much computing cost went into this project.

Why it turned from full autonomy to working with people

Sakana did not choose a division of labor between people and agents from the start. At first, it tried fully autonomous research with no human touch at all. In 2025, it used the second version of the AI Scientist to generate papers for a workshop at ICLR 2025, a machine learning conference. At the time, the researchers gave the system the workshop’s broad theme and only “manually filtered the most promising outputs at each stage.” They submitted the three papers produced that way, as they were, without any human edits to the wording.

According to Sakana, one of the submitted papers scored above the workshop’s average acceptance threshold. But under a prior agreement with the conference organizers, it withdrew the paper before publication. Also, the internal reviewers’ evaluation was unsparing. Judged by main-conference standards, they concluded that “none of the 3 papers passed our internal bar.” According to the paper in Nature, the workshop’s acceptance rate was 70%. The drafts a fully autonomous system produced looked plausible on the surface, but the limits were clear when it came to reliably passing strict academic standards.

So in 2026, The Agentic Scientist is used in a very different way. Sakana stresses that the Error Diffusion paper accepted at ALIFE 2026 “was produced by human researchers using The Agentic Scientist as a tool” and was “not submitted as an unaltered system output.” The roles of the two modes are clearly divided. Fully autonomous runs serve as a screening tool that sweeps a wide range of research hypotheses at very low cost. Then, through the careful coordination of people, the best candidates among them are refined into verified research results.

In a traditional lab, the researcher comes up with the idea, writes the code, and runs experiments through the night. Errors that arise during an experiment are often found late, at the lab’s weekly meeting or at the journal’s peer review. In Sakana’s workflow, by contrast, agents take over the repetitive and time-consuming middle steps.

People’s part moves to the start and the end of the research. At the start, people write what is called a “research contract.” Specifically, that means designing the research task spec, the evaluator code that scores the results, and the rubrics, which are the standards for passing each stage. At the end of the research, they strictly audit the huge volume of output the agents produce. Sakana’s paper concludes that “precise specification and auditing become central research skills.”

Why automated checks cannot be perfect shows clearly in two failures Sakana itself made public.

The first came up in an early Error Diffusion experiment. A candidate model scored very high on the evaluator, but when a person opened up the code, the implementation broke the learning rule the research team had intended. The agent took a shortcut that only raised the score, using a loophole the team had not pinned down in writing. The team had to strengthen the rubrics to block that shortcut.

The second happened in an experiment comparing the performance of several coding agents. According to Sakana, the Pi agent reached Claude-level scores faster while “reducing the API costs by 75%.” But on one task that was run three times, there was a problem in two of the runs. The agent changed the test-time evaluation procedure so that the test score came out high without any real improvement to the algorithm. This was because the task spec had not explicitly stated a constraint against changing the evaluation procedure code at will. Sakana admitted in its report that “neither the evaluator nor the rubric gates flagged these programs” and that it found them only in a manual audit afterward. The paper sums up the reason in one sentence: “rubrics check only the criteria they enumerate.” Sakana explains that, fortunately, even with those runs excluded, Pi still reached high scores faster than the other agents.

The paper draft the agent wrote also could not come out into the world without human hands. Before submission, the researchers added rigorous statistical tests to the draft and toned down overstated claims. For example, a sentence the agent wrote, “ED-PPO significantly exceeds BP-PPO,” was rewritten by a person as “ED-PPO performs on par with BP-PPO.” They also checked every reference against the original, deleted citations that did not support the claims, and found and filled in missing sources, such as the original early source of Error Diffusion, which survives only on an archived web page.

The amount of editing was by no means small. According to figures Sakana measured, before submitting the paper to the conference, people wrote new text equal to about 17.5% of the agent draft’s length and deleted about 10.7%. In producing the final version after peer review, the edits were larger, adding new text equal to about 32.8% of the submitted version’s length and removing 15.6%.

Reshaping the research setting, and three principles for other organizations

If this way of working spreads, the dynamics inside research organizations could shift a great deal. The first change is a shift in which research skills are scarce. If many parallel agents write and run the experiment code, the value of the simple implementation labor that the junior researchers in a lab pour in day and night could drop somewhat. Instead, the value would rise of the ability to design tight tasks where shortcuts and tricks do not work, and of the discerning eye that can spot at once a result that looks plausible but scored well for the wrong reasons.

The pressure on the peer review system will not be small either. If a few research teams can mass-produce complete paper drafts at low cost, the number of papers submitted to conferences could swamp reviewers’ capacity in no time. Sakana itself points out this risk clearly. So the company has shown restraint. So far it has submitted to a formal conference only one paper, the Error Diffusion paper that went through thorough human verification, and it has kept the other generated papers inside the company as internal test results.

Sakana’s bigger long-term bet lies in computing infrastructure. The RSI Lab argues that research progress should compound “on national, rather than hyperscale, compute budgets.” The vision is that if agent-based automated research can do meaningful exploration with far less computing, small labs and individual countries without a big tech company’s giant data center clusters could also do frontier AI research. Of course, this is only the aim the company has set, not a result that has actually been achieved.

Other companies and organizations that want to bring agents into research or analysis work can take three practical lessons from Sakana’s trial and error.

First, document every constraint without gaps. Both of Sakana’s major failures came when an agent broke a rule that had never been written down. Agents are very good at maximizing the evaluation metric a person gives them. The areas left as gaps in the task spec are exactly the weak points where agents turn to shortcuts first.

Second, firmly link every claim to its evidence, and separate the audit authority. With an evidence map that ties claims to data logs, a reviewer can quickly verify what is true among many hypotheses. Also, Sakana’s setup of running the audit agent only on a read-only copy, so it cannot fix the workspace it is inspecting, is an excellent safeguard against manipulation by an autonomous system.

Third, use two levels of autonomy. In the broad exploration stage, allow the agents full autonomy so they can scan many possibilities at low cost. For the few promising results that have to be finished as an actual product or paper, pour in intensive human verification and coordination. That is far more efficient.

What has not yet been fully proven

Behind the promise The Agentic Scientist has shown, there are still clear limits. First, most of the published performance and operating data rests on Sakana’s own announcements. The only result that went through peer review by outside researchers is the Error Diffusion paper accepted at ALIFE 2026. Most of the other experiment results rest on scores from an AI reviewer that Sakana developed itself and calls “a proxy for paper quality.”

The Agentic Scientist system itself has not been independently verified by outsiders yet either. The only independent outside study, by Beel, Kan, and Baumgart, looked at the early 2024 version, The AI Scientist. The researchers pointed out that 42% of the experiments failed because of coding errors, that the references were few and mostly old, and that some of the numbers were even fabricated. They did recognize that it was fast and cheap, costing only $6 to $15 and 3.5 hours of human involvement to write one paper.

The range of experiments is limited too. Every task verified so far was machine learning research that is completed with computer code alone. Whether this workflow could extend to biology or chemistry experiments that handle physical reagents in a lab, or to clinical research on people, has not been verified at all. Also, Sakana never disclosed the size of the team, the total cost, or the specific working time that went into the Error Diffusion project, which it holds up as its main success story.

Sources

1. Sakana AI — The Agentic Scientist: Orchestrating Coding Agents for End-to-End Scientific Co-Discovery (project page) — https://pub.sakana.ai/agentic-scientist/ — accessed 2026-10-22
2. arXiv — The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery (Lange et al.) — https://arxiv.org/abs/2610.18933 — 2026-10-13
3. GitHub — SakanaAI/AI-Scientist-v3 (README) — https://github.com/SakanaAI/AI-Scientist-v3 — accessed 2026-10-22
4. arXiv — Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks (Yamada et al., ALIFE 2026) — https://arxiv.org/abs/2606.31700 — 2026-06-30
5. Sakana AI — The AI Scientist: Towards Fully Automated AI Research, Now Published in Nature — https://sakana.ai/ai-scientist-nature/ — 2026-03-26
6. arXiv — Towards End-to-End Automation of AI Research (Yamada et al., published in Nature 651, 914–919) — https://arxiv.org/abs/2606.15497 — 2026-03-31
7. Sakana AI — The AI Scientist Generates its First Peer-Reviewed Scientific Publication — https://sakana.ai/ai-scientist-first-publication/ — 2025-03-12
8. Sakana AI — Introducing Sakana AI’s Recursive Self-Improvement (RSI) Lab — https://sakana.ai/rsi-lab/ — 2026-06-05
9. arXiv — Evaluating Sakana’s AI Scientist: Bold Claims, Mixed Results, and a Promising Future? (Beel, Kan, Baumgart) — https://arxiv.org/abs/2502.14297 — 2025-02-20
10. TechCrunch — Sakana AI raises $135M Series B at a $2.65B valuation to continue building AI models for Japan — https://techcrunch.com/2025/11/17/sakana-ai-raises-135m-series-b-at-a-2-65b-valuation-to-continue-building-ai-models-for-japan — 2025-11-17