The FinOps Layer for LLMs: Routing, Caching, and Metering That Pay for Themselves

There's a moment in every agent platform's life when the model bill stops being a rounding error and becomes a line item the CFO asks about. For most teams that moment arrived sometime in the past year. Agent workloads multiplied token consumption in a way chat never did, because an agent doesn't send one prompt. It sends forty, recursively, with tool outputs stuffed into context.

The good news is that this is now a solved-enough problem that the numbers are boringly consistent across the 2026 literature. Teams stacking smart model routing, semantic caching, and workload-aware batching report combined savings of 47-80%, with routing alone cutting 40-85% of spend on mixed workloads. The bad news is that the same reports agree on the failure mode. Teams that deploy routing without a quality benchmark in place degrade user experience and don't notice for weeks, because the system that would have noticed is the system they skipped building.

This post is about the layer underneath the savings: what has to be true of your platform before the optimization techniques are safe to turn on.

Metering first, optimization second

Every optimization decision, whether it's routing a request to a cheaper model, serving one from cache, or batching a job overnight, is a bet that the cheaper path is good enough. You can only settle that bet if you can measure the cost and the quality of every request and attribute it to its source. Which means the real foundation of LLM FinOps is the usage event, not the router.

In our own platform, every agent run and workflow execution publishes structured events onto an internal event stream (any durable stream works here; NATS JetStream and Kafka behave identically for this purpose). A dedicated metering consumer turns those into usage_events: tokens in, tokens out, model, latency, cache disposition, and, critically, the full attribution chain of tenant, agent, workflow, and step. Cost calculation is a downstream fold over that stream rather than a monthly reconciliation against provider invoices.

Getting this right buys you three things at once. The first is attribution. "The bill went up 30%" is a mystery. "Tenant X's enrichment workflow went up 30% because a prompt change doubled its average context length" is a fix, and per-tenant, per-agent, per-step cost is the difference between the two. The second is billing. If you charge customers for usage, metering is the product; the same event stream feeds invoices and dashboards, so there's one source of truth and finance and engineering stop arguing about whose number is right. The third is safety rails. Budget limits per tenant, per agent, and per run are only enforceable if spend is computed in near-real-time. A runaway agent loop that burns $400 before the nightly rollup notices is a metering failure, not an agent failure.

The cheapest token is the one you can explain. Optimization without attribution just moves spend around in the dark.

Routing: the 80% that's easy and the 20% that isn't

Model routing, which classifies each request and dispatches it to the cheapest model that can handle it, is where the headline savings live. The mature gateways now route across two dozen models and multiple providers with cost-focused, balanced, and quality-focused modes, and the easy wins are genuinely easy. Development and staging traffic has no business touching frontier models (that alone is typically around 15%). Bulk classification and extraction runs happily on small models. Only the genuinely hard reasoning tail needs the expensive tier.

The harder 20% comes in three parts.

Routing needs an eval harness, not vibes. The router's claim of "no visible quality loss" is an empirical claim about your workload. Before enabling a cheaper route for a task class, you need a benchmark for that task class, scored automatically and re-run whenever the route or the models change. Teams that skip this learn about quality regressions from customers.

Fallback chains are routing too. Provider incidents are now routine operational weather. A per-agent fallback policy, with a primary model, ordered alternates, and a rule for what quality floor justifies failing loudly instead of falling back silently, belongs in configuration rather than in incident-day heroics.

And route at the step, not the agent. Inside a single run, the planning step may justify a frontier model while the ten extraction steps behind it don't. Platforms that can only set one model per agent leave most of the routing savings on the table.

Semantic caching: free money with a threshold problem

Exact-match prompt caching is table stakes, and providers increasingly do it for you. The interesting margin sits in semantic caching: embed the query, search for a previously answered near-neighbor, and serve the cached response above a similarity threshold. At a realistic 40% hit rate it takes roughly 20% off the total bill.

Its sharp edge is the threshold. Too loose, and users get answers to almost their question, a quality bug no error rate will surface because nothing errored. Too tight, and the hit rate collapses to exact-match levels and the cache is dead weight. Two disciplines keep it honest. Cache disposition goes into the usage event (hit, miss, near-miss with similarity score), so cache behavior can be analyzed per task class instead of guessed at globally. And cache hits flow through the same evaluation sampling as live responses, so a cached answer that scores badly is evidence the threshold is too loose for that task class. Thresholds should be per-class, not global.

One hard rule from the multi-tenant world: the cache is tenant-scoped, always. A semantic cache shared across tenants is a data leak with excellent latency.

The frontier, simulated

To make the routing trade concrete we simulated it: a 100,000-request workload (60% bulk extraction, 25% assist/chat, 15% hard reasoning) priced against three models with realistic relative prices and per-class pass rates, computed as exact expectations. The gate is exactly the policy described above, and it's genuinely trivial once the benchmark exists:

for c in classes:                       # per task class
    floor = pass_rate["frontier"][c] - 0.02
    for m in ("small", "mid", "frontier"):     # cheapest first
        if pass_rate[m][c] >= floor:
            route[c] = m
            break

To be clear about epistemics: the magnitudes follow from the assumed mix and price sheet (both in the repo), so this demonstrates the mechanism rather than predicting your bill. The shape is the finding, and it's sharply asymmetric. The first 62% of savings costs 1.2 quality points. The last 36% costs six more, and here's the part that justifies this post's warnings: the ungated policy's damage sits almost entirely in the hard-reasoning class, which collapses from 95% to 62% pass rate while the blended average politely reports a 7-point dip. A dashboard averaging over all traffic would show a healthy-looking number as your hardest users' experience fell apart. Per-class benchmarks are what the gate consumes, and per-class is the only honest way to read quality. Code and verbatim results are in our repo's experiments directory.

Making it a loop, not a project

The reason we frame this as a FinOps layer rather than a bag of optimizations is that the pieces compound only when they're wired into a loop. Metering attributes spend, attribution reveals the expensive task classes, routing and caching target those classes, evaluation verifies quality held, and the savings get measured by the same metering that started the loop. Run once, it's a cost-cutting project that decays as workloads shift. Run continuously, it's a control system. Forecasting stops being an annual spreadsheet exercise too, because the event stream is the forecast input.

Our rough order of operations for a team starting from zero, ranked by return on effort:

1. Emit usage events with full attribution. Everything else reads from this.
2. Set hierarchical budget alerts, then limits. Alerts first; you'll be surprised what you find before you enforce anything.
3. Route the unambiguous traffic (dev/staging, bulk extraction) with a benchmark gate per task class.
4. Add semantic caching, tenant-scoped with per-class thresholds, and put cache disposition in the events.
5. Only then chase the long tail: batching, prompt compression, on-prem primaries for stable high-volume workloads.

The 2026 numbers say 60-80% reductions are achievable when the workload mix cooperates. Our experience says the teams that actually bank those savings, instead of trading them for silent quality debt, are the ones that built the metering spine first and treated every optimization as a hypothesis their evaluation layer had to confirm. Spend less, and prove it cost you nothing. That proof is the product.

ArthaVortex builds cost intelligence into agent platforms: metering, routing, and budget governance on one event spine. If your model bill is a mystery, we can help.