An AI model trained on public legal documents performed well. However, when applied to actual, consented customer contracts, its accuracy plummeted by 15 percentage points. This reveals the significant performance gap between clean, public training data and complex, private enterprise data.

A client used AI to create a tax step chart for a reorganization, but the AI incorrectly assumed the target was an S-corp. This fundamental error, which could have cost millions, was caught only by a human lawyer's judgment, underscoring the necessity of expert oversight.

After a document is drafted, lawyers ask an AI tool to review it for missed points or alternative angles. The tool acts like an impartial third party with vast analytical recall, offering suggestions that refine and improve the final work product at a minimal cost.

Beyond drafting documents, AI is highly effective at quality control tasks that humans often miss. Use it for proofreading, checking defined terms, and ensuring consistent formatting, which can catch subtle but important mistakes in complex agreements.

Resource-constrained startups are forgoing traditional hires like lawyers, instead using LLMs to analyze legal documents, identify unfavorable terms, and generate negotiation counter-arguments, saving significant legal fees in their first years.

Long-horizon agents are not yet reliable enough for full autonomy. Their most effective current use cases involve generating a "first draft" of a complex work product, like a code pull request or a financial report. This leverages their ability to perform extensive work while keeping a human in the loop for final validation and quality control.

While AI tools excel at generating initial drafts of code or designs, their editing capabilities are poor. The difficulty of making specific changes often forces creators to discard the AI output and start over, as editing is where the "magic" breaks down.

Messy AI-generated code ("slop") can still result in a functional product, hiding imperfections from the end user. In knowledge work, a slightly "off" AI-generated contract or memo creates immediate legal or business risk, as there is no interface to abstract away the sloppiness.

In-house lawyers don't need a perfect, infallible AI; they need a partner to manage an unsustainable volume of work like thousands of contracts. AI succeeds by augmenting human lawyers, allowing them to identify and focus on the most critical risks more efficiently.