AI for legal teams
The failure that sank the first lawyers to use AI was the invented citation. We build systems where every answer carries the document and clause it came from, and where a citation that does not resolve against a real document is never shown.
Key takeaways
The failure mode specific to this sector is the invented citation: a perfectly formatted reference to a document that does not exist. You prevent it with post-generation verification, not with an instruction asking the model not to do it.
The architectural rule is simple and not negotiable: a citation that does not resolve against a real document in the index is never shown. The answer is discarded and the query escalates.
The cases that hold up are about document volume, not judgement: search across the contract archive, review against a template, extraction of dates and obligations, and a first pass on due diligence.
It does not replace a lawyer’s judgement and we do not pitch it that way. It replaces the hours spent finding which contract a clause ended up in — work that today falls to the most junior lawyer in the firm.
The four cases that hold up
They all have the same shape: a lot of documents, an answer that is written down somewhere, and somebody currently looking for it by hand.
- "Which contracts have an exclusivity clause?", "which ones auto-renew?". It is the most frequent query in a legal team with several hundred live contracts, and today it gets answered by asking whoever has been there longest.
- Comparing an incoming contract against the approved template and flagging which clauses changed, which are missing and which were added. It does not decide whether the change is acceptable: it marks where to look.
- Expiry dates, notice periods, payment milestones, reporting duties. Extracted into a table with the source clause beside each row, and the alerts come off that.
- Across a batch of documents, classifying and flagging what needs human reading. It reduces the volume that reaches the lawyer; it does not replace their reading.
The invented citation, and how you actually close it
The case that went around the world — lawyers sanctioned for filing non-existent case law — was not a faulty model: it was a system without verification. A language model produces a real reference and a plausible one with equal fluency, because the format of a citation is an easy pattern to imitate. Asking it in the prompt not to invent reduces the frequency and does not eliminate the problem, and in legal work a low rate is still unacceptable.
You close it in the architecture. Every passage in the index carries a stable identifier — document, version, clause or article — and the answer has to carry it through. After generation, each citation is checked against a real passage; one that does not resolve invalidates the answer, which is never shown. The system escalates instead of gambling. It is a cheap check and it is the difference between a tool a lawyer can use and one they cannot.
The second decision is not letting the model write about what it did not retrieve. If the question has no support in the archive, the correct answer is to say it is not there, not to produce a reasonable-sounding paragraph. That gets measured: the evaluation set includes questions whose correct answer is "this does not appear in the indexed documents".
The numbers behind a legal system
The evaluation set is built from real queries from the team, with the correct answer validated by a lawyer at the firm. Without that validation there is no measurement, only an impression.
| What is measured | Threshold we propose | How it gets fixed |
|---|---|---|
| Invalid citation rate | Zero. No answer carrying a reference that does not resolve against a document in the index. | Post-generation verification. An answer with a broken citation is discarded and escalated, never shown half-corrected. |
| Clause-level recall@k | Agreed with the team before building, measured at clause level rather than document level. | Hybrid search: legal language has exact terms that semantic similarity alone does not separate. |
| Date extraction precision | Against a hand-validated batch, how many expiry dates and notice periods come out correct. | Structured output with schema validation, and the source clause mandatory on every row. |
| Correct refusal | Questions whose correct answer is "this does not appear in the indexed documents". | A higher confidence threshold and escalation. In this domain, silence is a valid answer. |
The evaluation set ships in an open format and belongs to the firm. It verifies the system without us, and it grades any other vendor by the same yardstick.
We do not build systems that issue legal opinions, draft pleadings or recommend litigation strategy. That is not a technical limitation: it is that the output would be an opinion with nobody accountable for it, and in this profession accountability has a name on it.
We are also not the answer if the archive is fifty contracts in a shared folder. This pays from several hundred live documents upward, with versions, annexes and amendments coexisting — which is when finding the right clause already costs billable hours.
Related services
RAG systems
Chunking, reranking and hybrid search, evaluated with recall@k and NDCG.
View serviceProject rescue
The system is already in production and answers badly. We measure it and determine what to fix.
View serviceIntegrations and APIs
MCP servers against the systems already in operation.
View serviceAI consulting
Architecture, evaluation criteria and cost per query before writing code.
View serviceMore from the blog
Frequently asked questions
How do you guarantee it will not invent case law or clauses?
With post-generation verification, not with an instruction. Every passage in the index carries a stable identifier — document, version, clause — and the answer has to carry it through. After generation we check that each citation resolves against a real passage; if one does not, the answer is discarded and the system escalates rather than showing it. On top of that, the system can only answer from what is in the index: if the firm does not index case law, there is nothing to invent it from.
Does it replace a junior lawyer?
No, and anyone selling it that way is selling you a problem. It replaces one concrete task the junior does today: finding which document a clause ended up in, in minutes rather than hours. What it does not replace is the judgement of whether that clause serves you, how it gets negotiated or what risk it carries — which is where the lawyer’s value sits and where the system has nothing to say.
Do our clients’ contracts leave the firm?
Only if you accept that, and in this sector the answer is usually no. We deploy on Azure OpenAI or Amazon Bedrock inside the firm’s own subscription, so documents and queries never leave your infrastructure. In any configuration we use enterprise plans where data sent through the API is not used for training, and we can filter names and identifiable data before inference where the engagement requires it.
Does it work with scanned contracts?
It does, but the cost has to be stated. A scan needs optical recognition, and the quality of the extracted text caps everything downstream: if OCR swallows a negation, the system answers the opposite with complete confidence. So we measure extraction quality before indexing and report what share of the archive fell below the threshold. If much of the archive is poor scans, the right project starts with digitising properly, not with the assistant.
Can it compare a contract against our template?
Yes, and it is among the highest-return cases because the correct answer is verifiable. The system aligns the incoming document against the approved template and flags which clauses changed, which are missing and which were added, each with the text of both versions side by side. It does not rule on whether the change is acceptable: it marks where to look, and that decision stays with the lawyer.
How long does it take and how is it priced?
A first scope — search across the contract archive, for instance — usually takes six to eight weeks to production, with something running from week one. It is quoted with fixed scope, price and date in a proposal 48 hours after the first call. The price is driven by document volume, the state of the archive — scans cost more — and what integrations are needed.
What if we already bought a legal AI tool and nobody uses it?
It is a known pattern and it is almost never the model. The frequent causes are that the tool is not connected to the firm’s real archive but to a parallel repository nobody feeds, or that it gave a badly cited answer early on and lost the team’s trust, which is very hard to win back. The diagnostic takes a week, hands back the failures ranked by impact with each fix and its effort, and if it does not reach three actionable findings, it is not billed.
Book 15 minutes
Tell us what you are building, or what stopped working. You leave the call with a concrete answer: it can be fixed, it can be built, or it isn't worth it.