Intelligence Efficiency: Routing Reasoning, Not Just Models

While building the Lite and Pro model paths for Interaction, we kept running into a counter-intuitive result: a larger model and a longer reasoning budget do not automatically buy higher task efficiency. What decides efficiency is usually not which model you pick, but whether a given step needs to reason at all.

What we mean by intelligence efficiency

We use intelligence efficiency to name a practical systems question: for the same class of user task, how can a product spend fewer tokens, wait less, and pay less while still returning a reliable enough result?

The intuitive answer is to reach for a stronger model. But stronger models are slower and more expensive — and, as the measurements below show, on many structured tasks they are no more accurate than a small one. Often the thing worth routing is not the model but the reasoning itself.

Setup

We ran a small, repeatable, deterministic benchmark through DeepSeek’s OpenAI-compatible API: 8 tasks spanning calculation, extraction, logic, code-execution understanding, migration planning, instruction filtering, long-context contradiction detection, and JSON repair. Each task has a single decidable correct answer and is auto-graded, while we record the returned token usage, latency, finish reason, and a usage-based cost estimate. The output cap was fixed at max_tokens = 1024.

On two models — deepseek-v4-flash and deepseek-v4-pro — we ran two configurations each: default thinking on, and thinking explicitly off. Prompts, tasks, and grading were otherwise identical.

Results

The result was direct.

With default thinking, flash accepted 5/8 at an average score of 0.812, and pro accepted 6/8 at 0.833; pro’s cost per accepted task (457.6 micro-USD) was more than triple flash’s (149.9). Pro was slightly stronger and clearly less efficient.

With thinking explicitly off, both models accepted 6/8 at the same 0.896 average, and cost per accepted task fell to 31.5 and 99.8 micro-USD respectively — roughly 4–4.6× cheaper than the default. On this structured slice, lean execution improved quality and cut cost at once, while flash held about a 3× cost-efficiency edge over pro.

Mechanism: the reasoning budget crowded out the answer

The migration-planning task explained the most. With default thinking, both flash and pro spent the entire 1024-token output budget on reasoning, returned a finish reason of length, and produced empty content — a score of 0. With thinking off, the same task returned correct JSON straight away. The failure was not that the model could not solve it, but that the reasoning budget squeezed the final answer out of a finite output window.

But turning thinking off was not uniformly better. In the same run the logic task regressed: both models fell from a perfect 1.0 to 0.5, picking the wrong winner. The net effect was that both landed at 6/8 — thinking off fixed planning and broke logic. That is precisely the point: thinking is not a "more is better" knob but a choice to open or close per task type.

What this means for Lite and Pro

This reshaped how we think about product tiers. Lite should not read as a "low-intelligence" version; it should be the higher-efficiency default execution layer — fast, cheap, and good enough for extraction, formatting, rule filtering, and simple code understanding.

Pro is not merely "the entry point to a more expensive model" either. It should be reserved for uncertain, long-horizon work where deeper thought has real marginal value — not applied by default to every step.

Conclusion

So a routing system cannot only ask "flash or pro?" It has to answer at least three questions at once: does this step need reasoning, does it need a stronger model, and is it worth spending more tokens? Only by separating those variables can a system find a real Pareto frontier across quality, cost, and latency.

Our conclusion: intelligence efficiency comes from routing reasoning, not just routing models. A good AI product system does not always call the strongest model — it knows when intelligence should be cheap, fast, and sufficient, and when deeper reasoning is genuinely worth paying for.

Intelligence efficiency comes from routing reasoning, not just routing models.

Limitations

This is a small, deterministic, auto-gradable task set, chosen deliberately to isolate one variable — the thinking switch. It is not a human-preference evaluation and does not cover open-ended writing or multi-turn agent work. Absolute numbers will move with model versions, pricing, and prompts. What is worth keeping is not any single figure but the shape: on structured tasks, lean execution tends to sit at the better point on the quality–cost curve.