Engineers from NVIDIA, Cognition, and OpenRouter sat down for 48 minutes at AI Engineer World's Fair and explained how model routing works in production better than any blog post on the subject.
This is what they told the room:
1. Cheaper per token isn't cheaper per task.
The panel kept returning to the trap teams fall into: picking the model with the lowest per-token price. Alex Atallah of OpenRouter did the math out loud.
"If you run terminal bench on Opus and Haiku, Opus does about three times better at one-tenth the cost of Haiku, even though Haiku is significantly cheaper per token."
A pricier model that finishes the task can cost you less than a cheap one that loops.
2. Let the smart model plan, then hand off the work.
Cognition's Fusion router keeps a frontier model in charge of the thinking and pushes execution down to a cheaper one. Walden Yan explained why that gets you more, not less.
"Because you're delegating the work to another model, you can let that model go at the task with much more depth. You can spin off three sub-agents to explore the codebase."
Delegation buys you more exploration for the same budget, not just a smaller bill.
3. No single model wins every task.
Tanay Varshney works on model evaluations at NVIDIA and pushed back on chasing one benchmark leader.
"It's not necessary that model A, if it scores higher on a coding benchmark, is better at every task there is. Models are strong at different things, rather than there being one model to rule them all."
Routing starts with knowing each model's jagged edges, not its leaderboard rank.
4. Naive task-based routing breaks the moment agents get involved.
The obvious approach is to send each task type to whichever model benchmarks best at it. Walden Yan called that fragile.
"Naive routing based on task type is extremely fragile, especially the more agentic the task is. The complexity and the type of task change over time, and you don't want to be left with a subpar model for the task you're now on."
In a real agent session the task shifts under you, so the routing has to shift with it.
5. Ignore the million-token context window.
Every provider advertises huge context limits. Alex Atallah told the room not to trust them.
"I'd never recommend using these models past 200k tokens, under 100k if you can. The intelligence falls off a cliff at some point."
That million-token limit is marketing. Your real working budget is far smaller.
6. Route by privacy, not just cost.
Carter Abdallah pointed to a use case most routing talk skips: keeping sensitive data off the cloud.
"Can you detect that my prompt has sensitive information? And if so, do that on device."
The router isn't only a cost lever. It decides what your data ever leaves the machine.
Watch the full panel, then read the guide on harness engineering below.
Video
Alex Prompter (@alex_prompter)
Article
harness engineering 101
On July 9, OpenAI launched ChatGPT Work. Within 48 hours, an agent running its flagship GPT-5.6 Sol model wiped most of AI investor Matt Shumer's home directory. The cause was a shell variable parsing
— https://nitter.net/alex_prompter/status/2077774842649247903#m