Building an AI cost governance layer for a staffing desk

Why does a staffing desk need this at all?

Because the workload is spiky and unbounded. A vendor blast can put ninety postings into the ingestion queue in an hour, each triggering extraction, embedding, and dedup comparison. Nothing about that is predictable from headcount or seat count, which is exactly what a subscription price assumes.

If you intend to charge for compute at cost, as we do, the requirement becomes stricter still: the number on the invoice has to be defensible line by line. That is not achievable by looking at a provider bill at month end and dividing.

What does per-action attribution mean in practice?

Every model call carries a tenant id, a job id, and an action name before it is dispatched, and those travel with the response and into the cost ledger. Not sampled, not reconstructed from logs afterwards — attached at the call site.

The reason to insist on this is that retrofitting attribution is close to impossible. Once calls are anonymous in the provider’s billing, no amount of log analysis will tell you that Tuesday’s spike was one tenant’s vendor blast rather than a bug.

It also changes the conversations you can have. "Extraction cost you $41 last month across 1,900 requirements" is a discussion about value. "Your AI usage was $610" is a discussion about suspicion.

How does tiered model routing work?

Default to the cheapest capable model and escalate only on low confidence. Most agent work is not hard — parsing a well-formed requirement, classifying a reply as interested or not, extracting a rate from an email — and a small fast model handles these correctly the overwhelming majority of the time.

The minority that genuinely needs a larger model is real: a mangled requirement with implicit skills, an ambiguous reply, a résumé whose relevance is structural rather than lexical. Paying frontier prices for the easy majority in order to cover that minority is how AI budgets quietly become the largest line on the invoice.

The critical detail is that escalation must be scoped. A low-confidence rate field should re-run the rate extraction, not the entire twenty-field document at the top tier. Per-field confidence is what makes that possible, and a single document-level score is what makes it impossible.

Why put this in a gateway rather than in each agent?

Because policy that lives in six agents is six policies that drift. One gateway gives a single place to change the default tier, add a model, cap spend per tenant, or fail over when a provider degrades — with no agent code touched and no redeploy.

It is also the only honest place to meter. A call is attributed before it leaves the gateway, which means the cost ledger and the actual dispatch cannot disagree.

• One default tier, changed in one place.
• Per-tenant token buckets and spend caps enforced before dispatch.
• Provider failover without agent redeploys.
• Attribution per tenant, job, and action — the basis for the cost dashboard.
• Full request tracing, so a bad extraction can be debugged after the fact.

How should spend caps behave when they are hit?

Degrade, do not stop — with one exception. A tenant hitting a soft cap should drop to the cheapest tier and keep working, with the condition surfaced to an admin immediately. Silently halting a pipeline mid-shift is a worse outcome than a slightly worse extraction.

The exception is a hard cap, which should stop dispatch entirely and page someone. The purpose of a hard cap is to bound the damage from a runaway loop, and a runaway loop that degrades gracefully is still a runaway loop.

Token buckets belong per tenant rather than globally, or one agency’s vendor blast becomes another agency’s outage.

What do retries do to cost?

Double it, unless every call carries an idempotency key derived from the work item. Agents are designed to be idempotent and safe to retry, which matters more once a gateway sits in the path: a step retried after a timeout should reuse the completed result rather than paying for the same extraction again.

The subtle version of this bug is worse than the obvious one. A retry that succeeds produces correct output and a duplicate charge, so nothing looks broken and the cost per action drifts upward with no visible cause.

Keys should derive from the work item, not from a request id, so a retry after a process restart still matches.

Where does caching actually help?

On the repeated parts of prompts rather than on results. Agent prompts carry substantial fixed context — the extraction schema, the field definitions, the classification taxonomy — and that prefix is identical across thousands of calls. Caching it is the single largest cost lever available after tiering.

Result caching is more dangerous than it looks. Two requirements that look identical may differ in a field that matters, and serving a cached extraction for the wrong one produces a confident, wrong record with no error anywhere.

What should the cost dashboard actually show?

Cost per action, trended, segmented by agent and by tenant. That is the view that makes a regression visible: extraction quietly moving from four cents to eleven because an escalation threshold drifted is invisible in a monthly total and obvious in a per-action trend.

Escalation rate deserves its own line. It is the leading indicator for cost, and it moves before the spend does — a rising escalation rate usually means input quality changed, not that the model got worse.

Admins should see this without asking. A cost governance layer that requires someone to request a report is a reporting layer, and it will be read once.

How does this connect to what a customer pays?

Directly, which is the point. Compute is metered to real usage and billed at cost plus a flat, itemised 12%, and the escalation policy is the reason "real usage" stays close to the cheap tier.

When cost is legible per action, charging for outcomes rather than seats stops being a marketing position and becomes arithmetic. It also means the incentive runs the right way: making the pipeline cheaper to run makes the customer’s bill smaller, and the placement fee is where the margin lives.

In what order should you build these four pieces?

Attribution first, before anything else and before any of it matters. It is the only piece that cannot be retrofitted, because once calls are anonymous in the provider’s billing there is no way to reconstruct which tenant or which action caused a spike. Build it while the call volume is small enough that nobody cares about the answer.

Idempotency second, because it is a correctness fix as much as a cost one and it gets harder to add once retries are load-bearing. A duplicate charge is the visible symptom; a duplicate side effect — a second submission, a second outreach email — is the expensive one.

Tiered routing third. It is the largest single cost lever, but it depends on per-field confidence being available, and per-field confidence is a change to how extraction is structured rather than a switch to flip. Attempting tiering before the confidence signal exists produces escalation policies driven by guesswork.

Caps and buckets last. They bound the damage from a failure mode rather than improving the normal case, so they are genuinely lower priority — right up until the first runaway loop, at which point they become the only thing anyone wants to talk about. Build them before the first paying tenant, not after.