Clause Combat Experiment:Manual vs. AI Playbook Generation

We are Ozeki’s legal interns this semester from the Start-up Legal Garage at the University of California School of Law, San Francisco (formerly known as UC Hastings). One of our tasks this semester was to run an experiment to benchmark how Ozeki’s software compares to the work of humans.The experiment was to take a few negotiated contracts and turn them into a detailed, clause-by-clause playbook matrix. We manually completed the same process as Ozeki’s software would – mapping out what got negotiated, how, and where. The hypothesis was as follows: if we manually did the same work thatOzeki did, would it be similar in terms of speed and accuracy?

The results were that Ozeki’s AI was as much as 60 times faster than manual review without any degradation of accuracy. What took us hours, Ozeki was able to do in minutes. Watching the software in action made its value crystal clear – and proved exactly why tools like this are so powerful and can greatly benefit the legal industry.

Nature of the Experiment

To get a fair comparison against Ozeki’s software, the experiment was structured in the following way. First, we started with abstractions of a contract dataset provided by one of Ozeki’s design partners, which we had scrubbed of all identifying information. Then, we gave each clause a quick proofread to make sure everything was readable and consistent.

From there, we built a matrix of negotiated terms – a manually created "playbook” that maps out the different negotiated choices across the contracts. But we didn’t stop there, we also redlined each clause against the gold-standard template language (basically, the ideal version for the client) to highlight exactly what “gives” were being made in each deal to “get”what the client wants.

At the same time, Ozeki’s AI read each contract and automatically stored the output in JSON files and a postgres database. Collectively, our human process took us over ten hours, including two conversations with the company’s CEO to manage the process. Ozeki’s software processed the same information and took 10 minutes, 8 seconds. A manual review of the accuracy of the playbooks showed no significant errors in either the manually or AI created versions.

High level Implications

Ozeki outperformed manual review in speed and consistency, but it also demonstrated the promise of other exciting features.  In the 10 minutes, the AI created a summary for each clause and an “importance weight.” This weight signifies how a new or negotiated clause aligns with or varies from the preferred template in terms of meaning and contractual risk. The importance weight can combine several signals from the data such as the language’s semantic distance from the template clause, frequency of appearance in negotiated contracts, pricing factors, and manual stack ranking. The use of an importance weight not only increases speed of negotiation, but it also highlights what the best trades in the negotiations might be. Down the line, this can evolve to include more formalized redlining capabilities, letting users see exactly what changed between iterations at a glance.

AI in Lawyering

As law students — and soon-to-be junior associates — we’d be lying if we said we weren’t at least a little nervous about AI tools. If software can do the work of a first-year associate, how does one get to be a second-year associate? Or worse, even get the first-year job in the first place? Where would we fit in?

Through this project, we realized AI isn’t replacing lawyers. Instead, it is reshaping what lawyers spend their time on. AI frees us from the kind of work that’s time-consuming, repetitive, and prone to error — the kind of work that junior and senior lawyers alike don’t want to do, and clients don’t want to pay for. No client wants to be billed ten hours for something a computer can do in ten minutes. And frankly, junior associates would rather not spend those ten hours doing it either.  One might ask whether this kind of work to build a strategy from the ground up might ever get completed, shortchanging the organization.

As much as AI might be a threat to the status quo, it also offers a profound opportunity.  Where we do bring value — and where we’ll continue to — is in applying judgment, negotiating trade-offs, and giving advice. Humans consider not just what’s in the contract, but what’s behind it. As we start our careers as attorneys, we prepare ourselves not by completing rote tasks, but by starting smarter, more prepared, and with a clearer, strategic view of what really matters in a deal.

About the Authors

Alysha worked as a legal analyst and compliance lead at the Opentensor Foundation, working at the forefront of open source AI, and is currently an LLM student interested in policy and legislation around decentralized AI.

Nicole graduated from University of Virginia and worked as a corporate paralegal prior to law school. She is a second-year law student interested incorporate finance, hoping to work with later-stage start-ups on their exit strategies.

‍