Crosby: Training Agents on Lawyers' Edits

Key points

• When a lawyer fixes an agent’s redline, the fix itself becomes training data.
• Lawyers and engineers work as one team, and lawyers write the prompts themselves.
• Crosby measures performance by total turnaround time and review time, not billable hours.

A law firm that reviews contracts for fast-growing tech companies

Every time a sales team tries to close a deal, one thing slows it down: the routine review of commercial contracts. A customer sends a draft contract. A lawyer reads it, changes the wording, and adds comments explaining why. Then the lawyer sends it back. The deal closes only after this is done.

Crosby is a New York law firm that specializes in reviewing and negotiating exactly these commercial contracts. The documents it handles most are nondisclosure agreements (NDAs), master services agreements (MSAs), and data processing agreements (DPAs). Its clients are mostly fast-growing tech companies such as Cursor and Clay.

Crosby opened in 2024. It was founded by Ryan Daniels, who worked as a lawyer at the big firm Cooley and served as a startup general counsel, and John Sarihan, who was an engineering lead at the fintech company Ramp. The company calls itself a “vertically integrated AI-native law firm.” It does not make software to sell to other law firms. It is the law firm itself, with licensed attorneys, malpractice insurance, and legal responsibility for its work.

According to what Ryan Daniels told Artificial Lawyer in June 2026, the company is split into two entities. One is the law firm that actually provides legal services. The other is a separate company that licenses the technology and handles non-legal work such as billing and sales. The company has raised about $86 million so far, most recently a $60 million Series B in March 2026.

How one contract goes in and comes out as a redline

The work starts when a client tags Crosby in Slack, the collaboration messenger, or sends an email. On Sequoia’s Training Data podcast in September 2025, John Sarihan described this first step: “Today we’ve basically built a paralegal agent, which is, like, in charge of routing all the work.” This agent assigns the incoming request to a lawyer who will review it.

Next, agents do the first pass. An agent summarizes the whole contract, proposes a redline, which is a markup that shows changes to a contract in red, and drafts comments that explain why each change was made. The founders say that explaining the reason for a change is the area where AI is strongest.

A licensed attorney receives the agent’s draft. The lawyer fixes and polishes what is needed, then sends the final version to the client. The founders said on the podcast that clients usually get the revised document back within a few hours.

For this flow to run smoothly, the system has to remember each client’s own principles. Crosby prepares a negotiation playbook agreed in advance with each client. On the company’s Cursor customer page, Crosby reviews contracts within a negotiation playbook agreed with Cursor beforehand. When an issue falls outside that standard, it passes the matter to Cursor’s senior legal counsel for a decision.

The client’s preferences and past negotiation decisions are stored in a database called Crosbrain. Each time a new contract arrives, the system automatically pulls from the database the client’s earlier contract versions, the counterparty’s past positions, and the priorities the client cares about most, and uses them in the work.

Agent drafts and lawyer edits build a training set

An agent drafts first and a lawyer edits, so Crosby ends up with two versions for every review. One is the redline the agent first proposed. The other is the redline the lawyer finally sent to the client. A company blog post from October 19, 2026 says: “Every one of those attorney decisions is a potential training label.” Crosby uses this comparison data in three ways.

First, it runs mechanical checks that need no human judgment. These automatically check whether an edit landed in the wrong paragraph, whether an explanatory comment is attached to the right wording, and whether the edits created a new term that is not defined anywhere in the contract. Crosby says these rule checks once caught a change that would have attached 40 malformed comments at once.

Second, to measure lawyers’ judgment, it has them compare two redlines instead of giving a score. When lawyers were asked to look at a redline and score it from 1 to 10, each person used a different standard, and the ratings were not consistent. So Crosby shows a lawyer two redlines, blinded so the lawyer cannot tell whether a model or a person wrote each one. It asks which one the lawyer would rather send to the client. Then it asks whether the lawyer could send the chosen one as it is, without touching it.

Third, it built a Verifier model that predicts the chance that a lawyer will accept a suggestion as it is. Crosby trained the Verifier on about 13,000 labeled redlines. The model’s goal is to calculate in advance the probability that a lawyer will send the agent’s redline to the client without changing it. The company plans to use this model to focus lawyers’ valuable time on the edits that really need high-level judgment, and to score the agents ahead of time on contracts no lawyer has reviewed yet.

Of course, this data has clear limits, and the company does not hide them. In one of its own experiments, two lawyers redlined the same 20 NDAs. Out of about 500 edits, they agreed on only 45%. Also, some of the context lawyers used when editing contracts never reached the agent, such as a preference the client mentioned in passing on a phone call.

The mix of people who build this system is unusual too. According to Ryan Daniels in a June 2026 Artificial Lawyer interview, the company then had about 45 lawyers and about 40 people in engineering and operations. In October 2026, the company website listed more than 50 licensed attorneys. Lawyers and engineers work in teams, and a lawyer and an engineer are often paired for each client. Ryan Daniels described the office layout this way: “desks staggered: lawyer, engineer, lawyer, engineer.”

Lawyers do more than pass requirements to the AI engineers. They write the prompts themselves. On the podcast, Daniels stressed, “Get them writing prompts.” The internal evaluation tool is built so both groups look at it together. It opens a real Word document and shows the agent’s redline right next to the redline the lawyer is expected to make. Lawyers can edit the client’s background context, and engineers handle the system prompt and the underlying tools. That is how they divide the work.

Lawyers as editors, and a fee per contract

At a traditional law firm, a lawyer writes the contract redline from scratch and bills an hourly rate for the time the work took. At Crosby, an agent makes the first draft, and the lawyer works as an editor who reviews and polishes it. The lawyer’s edits then become valuable training data that teaches the agent again. According to Daniels, the firm has no so-called “super senior” partners, and profits are reinvested in technology research instead of being paid out to partners.

The standard for judging work is different too. Crosby does not tally the billable hours lawyers work. John Sarihan said, “we run on total turnaround time.” Total turnaround time means the total time from when the contract is received from the client to when the final redline is handed back after several rounds of review. The company also watches a metric called HuRT, the time humans spend directly on review. It adds a third metric that checks whether the finished work matches each client’s appetite for risk. Lawyers are evaluated on how much they contribute to turnaround time and review time, not on billed hours.

According to the results of controlled tests that Crosby released, after the agent was tuned for the job, lawyer review time fell by 74% on one client’s NDAs and by about 80% on another client’s MSAs. These are figures the company measured itself.

The fee structure also differs from the legal profession’s habits. Crosby charges a fixed fee for each contract document, not by the hour. The company’s homepage carries the line “Don’t buy hours. Buy outcomes.” Daniels says, “The fundamental unit is a contract.” The problem is that the firm must set the per-contract fee in advance, without knowing how many rounds of negotiation with the other side will follow. In the interview, Daniels admitted that its forecasts sometimes miss, and “we lose money on work all the time.”

According to an analysis by the independent research firm Contrary Research, traditional law firms charge similar amounts to Crosby for basic contract review. In its view, what sets Crosby apart is not a low price but fast turnaround and a flat fee billed only once per contract.

So how far has automation come? For now, lawyers check every output themselves. In a post quoted by Artificial Lawyer in July 2026, Daniels wrote, “Today, our lawyers review every single work output.” In the same post, he said the firm plans to buy professional liability insurance for the agents themselves, and that in the future this kind of human review “won’t be necessary.” But Bloomberg Law reported that across the legal industry, having lawyers review AI work directly is still the dominant rule.

Billable hours under pressure, and three lessons for expert organizations

If the way Crosby operates holds, long-standing habits in the legal industry could be shaken. First, hourly billing could lose strength in high-volume, standardized work. In an environment where software does most of the drafting, a traditional firm that bills by the hour runs into a contradiction: the faster it finishes the job, the less revenue it earns.

Crosby turned this around. Because it charges a fixed fee per contract, the faster the review, the larger the margin per contract can be. If the effort needed per document becomes predictable, routine contract review could gradually move from hourly billing to flat per-contract fees. But complex matters, where the scope is hard to see in advance, will probably stay hourly. The founders admitted this on the podcast too.

The firm’s characteristic pyramid structure could also change. On the podcast, the founders compared traditional law firm training to a master teaching an apprentice. If agents take over the drafting, the hands-on work that new lawyers once learned from could shrink or change in nature. Crosby itself plans to build agents in stages, modeled on the roles of junior associates, senior associates, and junior partners. That leaves a basic question: if future lawyers never draft anything themselves, how will they build legal judgment?

Companies in other fields that need expert review can find three lessons in Crosby’s review system.

First, save both the agent’s draft and the expert’s final edit. If both versions are kept side by side, everyday work itself becomes training data that teaches and evaluates the AI, without a separate data-labeling project.

Second, run mechanical rule checks first. Even in work that needs deep professional judgment, there are areas that can be checked mechanically, such as where a paragraph sits or how a term is defined. If rules filter these out beforehand, experts do not waste time catching small errors.

Third, have experts compare two options directly instead of giving vague scores. If experts use different scoring standards, the data becomes less reliable. Instead of asking for a number from 1 to 10, asking “Which of these two would you send?” can produce much more consistent and useful data.

What still has to be proven

Keep in mind that the efficiency and turnaround-time figures Crosby presents were all measured by the company itself in a controlled environment. Whether the same time savings hold up in real outside work under many different conditions remains to be seen.

Also, because lawyers now check every output one by one, lawyer labor cost rises as contract volume grows. Whether an insured agent can handle contracts on its own, without a lawyer’s final review, has not yet been shown. Bloomberg Law pointed out that lawyers and regulators still question what accidents such AI-specific insurance could actually cover.

Finally, the work Crosby handles is mostly standard commercial contracts between tech companies. Whether this agent-centered model of collaboration can work in harder areas of law, where interests clash sharply and there are many variables, such as mergers and acquisitions (M&A), has not been shown.

Sources

1. Crosby — How we do evals at Crosby — https://crosby.ai/blog/how-we-do-evals-at-crosby — 2026-10-06
2. Crosby — Scaling Legal Judgment — https://crosby.ai/blog/scaling-legal-judgment — 2026-10-19
3. Crosby — Introducing the Crosby Intelligence Fellows — https://crosby.ai/blog/introducing-the-crosby-intelligence-fellows — 2026-10-13
4. Crosby — Series B and Planting our Flag — https://crosby.ai/blog/series-b-planting-our-flag — 2026-03-31
5. Crosby — Series A and Smarter Agreements, Faster — https://crosby.ai/blog/series-a-and-faster-agreements — 2025-10-08
6. Crosby — Introducing Crosby, the world’s first hybrid law firm — https://crosby.ai/blog/introducing-crosby-the-worlds-first-hybrid-law-firm — 2025-06-17
7. Crosby — homepage — https://crosby.ai — accessed 2026-10-22
8. Crosby — Cursor case page — https://crosby.ai/cursor/ — accessed 2026-10-22
9. Sequoia Capital — Deal Velocity, Not Billable Hours: How Crosby Uses AI to Redefine Legal Contracting (Training Data podcast) — https://sequoiacap.com/podcast/training-data-crosby — 2025-09-02
10. Artificial Lawyer — The Crosby Story – With Co-founder, Ryan Daniels — https://www.artificiallawyer.com/2026/06/18/the-crosby-story-with-co-founder-ryan-daniels/ — 2026-06-18
11. Artificial Lawyer — Crosby to Insure Its Agents for Legal Liability — https://www.artificiallawyer.com/2026/07/28/crosby-to-insure-its-agents-for-legal-liability/ — 2026-07-28
12. Bloomberg Law — AI Agent Insurance Points to Future of Law Without Lawyers — https://news.bloomberglaw.com/legal-ops-and-tech/ai-agent-insurance-points-to-future-of-law-without-lawyers — 2026-08-03
13. Contrary Research — Crosby — https://research.contrary.com/company/crosby — 2026-10-14