Twitter/X

Jessy Lin highlights a synthetic law firm, Calderwood & Harkness (C&H), as a…

2026-08-07 · 20:17 UTC ·@realJessyLin ·1 min read

Brief

Jessy Lin endorses LAB’s open-sourced synthetic law firm Calderwood & Harkness (C&H), built with engramlab, as a persistent, firm-context benchmark that lets models accumulate and reuse firm-specific knowledge; she contrasts this with single-instance agent benchmarks and credits Julio Pereyra, @nikogrupen, and @gabepereyra for the work.

Why it matters

Jessy Lin highlights a synthetic law firm, Calderwood & Harkness (C&H), as a persistent work environment benchmark that contains firm-specific work product and is being open-sourced as LAB’s next expansion, built in collaboration with engramlab.

Key details

  • She contrasts C&H with typical agent benchmarks that drop models into isolated task instances (e.g., “here’s a repo, fix this bug”), claiming C&H lets models compound firm-specific experience—like senior engineers’ mental maps or lawyers’ playbooks—to improve continual learning/memory.
  • She credits collaborators Julio Pereyra (@ItsJulioPereyra), @nikogrupen, and @gabepereyra and calls the project an exciting step toward better continual learning benchmarks.
Cleaned source text

I’ve been saying for a while that we need better benchmarks for continual learning/memory — I think this is a really exciting step towards this!

We’ve built an entire synthetic law firm (𝐂𝐚𝐥𝐝𝐞𝐫𝐰𝐨𝐨𝐝 & 𝐇𝐚𝐫𝐤𝐧𝐞𝐬𝐬 ⚖️), modeled after real legal work – a 𝘱𝘦𝘳𝘴𝘪𝘴𝘵𝘦𝘯𝘵 work environment where agents do many tasks over the 𝘴𝘢𝘮𝘦 underlying context. Most agent benchmarks are built in this world where a model is dropped into an independent task instance and has to do its best / explore as quickly as possible (e.g. "here's a repo, fix this bug"). But we want to move towards a world where models can build on their experience. A senior engineer who has a mental "map of the codebase" can pinpoint issues much more quickly and effectively. In the same way, lawyers build up experience, learning what arguments succeed in front of regulators, playbooks for dealing with certain cases, etc.

It's the 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘵𝘪𝘢𝘵𝘪𝘰𝘯 between law firms that makes the actual practice of law interesting. Every model (and law school student) knows a lot about law from studying the textbooks, but our goal is to build models that can compound and augment the rich, internal/private knowledge that makes firm A more successful than firm B.

now it's finally possible to understand and build towards that! there's a lot more work left to do, but we've been learning a lot from @ItsJulioPereyra @nikogrupen @gabepereyra to bring these models closer to real world tasks :)

Julio Pereyra (@ItsJulioPereyra)

Article

LAB: Law Firm Knowledge

We are open-sourcing our next LAB expansion: a synthetic law firm, Calderwood & Harkness (“C&H” or “the firm”). Built in collaboration with @engramlab, the firm contains work product from more than

— https://nitter.net/ItsJulioPereyra/status/2085772997944803682#m