Tiered models behind an AI gateway

Why tier models at all?

Most agent work is not hard. Parsing a well-formed requirement, classifying a reply as interested or not, extracting a rate from an email — a small fast model handles these correctly the overwhelming majority of the time.

A minority of calls genuinely need a larger model: a mangled requirement with implicit skills, an ambiguous reply, a résumé whose relevance is structural rather than lexical. Paying frontier prices for the easy majority to cover that minority is how AI budgets quietly become the largest line item on the invoice.

What triggers an escalation?

Per-field confidence, not a global score. The extraction step returns a value and a confidence for each of roughly twenty fields on a requirement. If the fields that matter — work authorization, tax term, rate, end client — come back below threshold, that requirement is re-run on the larger model.

The important detail is that escalation is scoped. A low-confidence rate does not force a full re-extraction at the top tier; it re-runs the part that was uncertain.

Why put this in a gateway instead of each agent?

Because policy that lives in six agents is six policies. Routing through one gateway gives a single place to change the default tier, add a model, cap spend per tenant, or fail over when a provider degrades — with no agent code touched.

It is also the only honest place to meter. Every call is attributed to a tenant, a job, and an action before it leaves, which is what makes a per-action cost dashboard possible at all.

• One default tier, changed in one place.
• Per-tenant token buckets and spend caps.
• Provider failover without agent redeploys.
• Attribution per tenant, job, and action — the basis for the cost dashboard.
• Full request tracing for debugging a bad extraction after the fact.

What about retries?

Agents are idempotent and safe to retry, which matters more once a gateway sits in the path. Every call carries an idempotency key derived from the work item, so a retried step after a timeout reuses the completed result instead of paying for the same extraction twice.

What does this buy the customer?

A bill you can predict and audit. Compute is metered to real usage and billed at cost plus a visible 12%, and the escalation policy is the reason "real usage" stays close to the cheap tier. When cost is legible per action, charging for outcomes rather than seats stops being a marketing position and becomes arithmetic.