Hybrid Token-Efficient Routing: Spend Tokens Only When a Proof Fails
A separate project: a Python routing agent that uses deterministic solvers first, then gated local inference, then a cloud fallback.
The Hybrid Token-Efficient Routing Agent is a separate Python project built for the AMD Developer Hackathon ACT II. Its operating principle is conservative: use zero tokens when a deterministic solver can prove an answer; use a local model only when a category-specific gate validates the output; and send the remaining work to a cloud model when quality needs that escalation (Arora).
The routing pipeline starts with JSON-schema validation and an ordered task classifier. It then asks a more important question than ‘which model is best?’: is there a deterministic proof path? For arithmetic, logic, constrained summarization, named entities, sentiment, factual extraction, and selected code tasks, a specialized solver either produces a validated answer or abstains.
Abstention is a feature, not a failure. The repository’s deterministic solvers are designed to decline ambiguous inputs instead of creating the false confidence that a cheap route is correct. That policy protects the next tier: a failed proof does not become a silent answer; it becomes an explicit reason to consider local or cloud inference.
The local tier packages a pinned llama.cpp runtime with a Qwen3.5-2B model inside Docker. It is useful only when the category is eligible and its output passes category-specific validation. This keeps a local model from being treated as a universal replacement for a larger model merely because it is cheaper or available offline (Arora).
Cloud routing is the final quality path, with runtime configuration for the Fireworks endpoint and allowed models rather than hidden provider assumptions. Batching, retries, validators, and a crash-safe output writer protect the evaluator contract: the system must always emit a valid results JSON array even when an upstream provider fails.
The project includes a synthetic 1,000-question development dataset, 282 passing tests reported by the repository, and diagnostic tooling that records the resolving tier without changing submission output. The accuracy/token frontier shown below is therefore not a decorative chart: it is project data used to examine the trade-off that the router is explicitly built to control.
