Twitter/X

At a PE-backed medical billing company processing claims for hundreds of…

2026-08-07 · 07:39 UTC ·@alex_prompter ·2 min read

Brief

A PE-backed medical billing company processing claims for hundreds of facilities had one co‑founder who alone understood the legacy schema; a senior AI engineer encoded her 20 years of expertise into an agent in six weeks. With a 1.5 FTE pod and $200–$300/month compute they built a 60-case eval harness (59/60 correct), penny-perfect denial reconciliation, and ~9–10 weekly hours saved.

Why it matters

At a PE-backed medical billing company processing claims for hundreds of healthcare facilities, one senior AI engineer encoded 20 years of the co‑founder's domain expertise into an agent in six weeks by translating her hand‑written SQL reasoning into scoped lookup rules, remittance pipelines, and a decision tree evaluation order.

Key details

  • The project ran as a 1.5 FTE 'AI Velocity Pod' (one senior AI engineer + a fractional AI delivery lead) with $200–$300/month in compute; an eval harness of 60 golden test cases and automated judges produced 59/60 correct on first deploy, 15/15 denial cases passing, and a flagship denial analysis reconciling to the penny.
  • The single critical condition for success was explicit knowledge capture from the sole domain expert; the author claims AI increases the premium on that expert and lets them compound value by reviewing and directing the agent, yielding roughly 9–10 hours saved per week.
Source evidence

The best example how eval harnesses are built in production

Mark Ajzenstadt (@mardehaym)

The only reason your company can answer questions about its own data is because one person hasn't quit yet.

At a PE-backed medical billing company processing claims for hundreds of healthcare facilities, one AI engineer encoded 20 years of domain expertise in six weeks.

Twenty years of data, a working rules engine, and one person who could answer any analytical question about the system: the co-founder, hand-writing SQL against a legacy schema. She was the only person who understood the data.

Inside the pod:
1. One senior AI engineer building the agent and encoding domain methodology
2. A fractional AI delivery lead on architecture, review, and quality gates

Total: 1.5 FTE. $200 to $300 a month in compute.

The old way: the co-founder ran denial analyses, financial health assessments, and payer mix questions by hand. Full working days on single research requests. She was the only path to the data.

The engineer spent the first three weeks writing zero code. Each session went to encoding how the co-founder reasons through queries. Denial definitions became scoped lookup rules. Remittance logic became a sequenced pipeline. Evaluation order became a decision tree the agent follows before output.

The eval harness came next: 60 golden test cases from the co-founder's actual workload, verified by her own SQL against production. Automated judges grade responses on correctness and grounding. Score of 1: the agent invented a number. Score of 5: the claim traces to the data. The team doesn't ship any release that drops accuracy.

After six weeks on the same data:

  1. 59 of 60 golden questions correct on first deploy
  2. Flagship denial analysis reconciles to the penny against the source database
  3. 15 of 15 denial cases pass automated grading
  4. CPTO challenged one answer; the agent self-corrected and matched
  5. Nine out of ten hours saved per week. $200 to $300 a month in compute.

The co-founder had been spending full working days on those same requests. Nobody else could navigate the schema.

One condition makes this work or fail. The knowledge capture can't be skipped.

Buyers describe the same bottleneck: their most valuable domain expert buried under manual queries nobody else can run.

AI raised the premium on domain expertise.

With an AI Velocity Pod, that person compounds: reviewing and directing while the agent handles volume, verified to the penny.

— https://nitter.net/mardehaym/status/2085631480777044433#m