Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 16

AI FinOps & Economics

Last updated · 8 min read

16.1 Token Economics & Cost-Per-Outcome

Priority: Must Understand

Executive Definition: Token economics is the unit-cost model of AI systems: providers bill per input and output token, so cost scales with prompt length, context window usage, output verbosity, and (for agents) the number of intermediate steps taken. Cost-per-outcome reframes this around the metric that actually matters to the business (dollars spent per successfully resolved ticket, per correctly extracted document, per completed task) rather than dollars spent per token or per API call.

Why It Matters: Cost per token is a weaker metric than cost per successful outcome because it says nothing about whether the money was well spent: a cheap model that fails and needs three retries, or a verbose model that uses more tokens per response but resolves the task in one pass, can have opposite cost-per-token and cost-per-outcome rankings. This gap matters far more for agentic workflows than for single-turn chat: an agent's cost is a function of an unbounded, data-dependent number of tool calls, retries, and reasoning steps, so the same task can cost 2x or 20x depending on how many loops the agent takes to converge: a degree of per-transaction cost variance that traditional SaaS spend, dominated by flat seat licenses and predictable compute reservations, simply does not have. FinOps for AI practitioners describe this explicitly as a shift from cost-per-unit-of-infrastructure to cost-per-unit-of-value, and it requires monitoring discipline closer to production incident response than to conventional cloud budgeting (finops.org).

What I Need to Understand:

  • Input tokens, output tokens, and (for some providers) cached-token discounts are billed at different rates: a system that reads a large document repeatedly per turn has a very different cost profile than one that generates long responses.
  • Cost-per-outcome requires defining what "successful outcome" means for each workflow (ticket resolved without escalation, extraction verified correct, task completed without human intervention): this is a product/business definition, not something the finance or engineering team can set unilaterally.
  • Agentic cost is fundamentally less predictable than single-turn chat cost: retries, tool-call loops, and multi-step reasoning chains mean the same input can produce wildly different token consumption run to run, which is why per-agent, per-workflow cost monitoring (not just aggregate monthly spend) is required.
  • A model that is more expensive per token but reaches a correct outcome in fewer steps can be cheaper per outcome than a "cheap" model that requires more retries or human correction: token price alone is not a valid basis for model selection.
  • Cost-per-outcome tracking depends on having the same tracing infrastructure used for trajectory evaluation and observability: cost and quality data need to be joined at the trace level, not analyzed separately.

Questions I Should Be Able to Ask My Team:

  1. Do we track cost per successful outcome for our top agentic workflows, or only aggregate token spend per model/application?
  2. What is the observed variance (not just the average) in cost per task for our agentic workflows, and do we have alerting on outlier-cost runs?
  3. When we compare models for a given workflow, is the comparison based on cost-per-outcome including retries and escalations, or on list price per token?

Technologies / Standards / Companies to Know: FinOps Foundation (FinOps for AI working group), Bedrock/Azure/Vertex usage and billing APIs, LangSmith/Arize cost-tagged tracing.

Recommended Learning:

Time Investment: 1 hour

16.2 Cost Controls: Model Routing, Caching & Smaller-Model Substitution

Priority: Must Understand

Executive Definition: These are the three primary engineering levers for reducing AI spend without simply refusing to use larger models: model routing (also called cascading) sends each request to the cheapest model capable of handling it, escalating to a larger model only when needed; caching (exact-match or semantic) avoids paying for a model call at all when an equivalent request has been answered before; smaller-model substitution deliberately uses a lower-cost, fine-tuned, or distilled model for well-scoped subtasks instead of a frontier model for everything.

Why It Matters: These controls sit directly downstream of cost-per-outcome logic: the goal is not the lowest token price but the lowest cost per successful outcome, so each of these levers has to be evaluated against eval and groundedness metrics, not deployed blind. A router or cache misconfigured to prioritize price alone will quietly degrade output quality in ways that don't show up until a customer complains or an audit occurs.

What I Need to Understand:

  • Model routing/cascading typically works by scoring task complexity (via a lightweight classifier or the smaller model's own confidence) and escalating only the subset of requests that need a stronger model: this is architecturally similar to the model-router feature now built into some cloud-native gateways (e.g., Azure AI Foundry's model router).
  • Caching for LLMs comes in two forms: exact-match caching (only helps with literally repeated queries) and semantic caching (matches queries that are semantically similar, not identical), which has a much higher hit rate but requires an embedding/similarity layer and careful tuning to avoid serving a cached answer to a subtly different question.
  • Smaller-model substitution works best for narrow, well-defined subtasks (classification, extraction, formatting) and worst for open-ended reasoning: the substitution decision should be scoped per task type within a workflow, not per application.
  • Every cost control here needs a quality guardrail wired to it: routing thresholds, cache hit criteria, and substitution boundaries should all be validated against golden-dataset and production-eval metrics, not tuned by cost alone.
  • These controls are typically implemented at the gateway layer, which is why gateway architecture choice and cost-control strategy are not independent decisions.

Questions I Should Be Able to Ask My Team:

  1. What is our model router's escalation criterion, and has it been validated against actual task failure rates, not just cost savings?
  2. What is our semantic cache's similarity threshold, and how do we detect if it's serving a stale or wrong cached answer to a similar-but-different query?
  3. For workflows using smaller-model substitution, what is the measured cost-per-outcome delta versus using the larger model for the same subtask?

Technologies / Standards / Companies to Know: Azure AI Foundry model router, LiteLLM/Portkey routing and caching, semantic caching (e.g., Redis-based implementations), model distillation/fine-tuning for task-specific substitution.

Recommended Learning:

Time Investment: 1 hour

16.3 Chargeback, Showback & Runaway Agent Cost Risk

Priority: Monitor

Executive Definition: Chargeback allocates AI spend directly to the business unit or application that incurred it, typically billed against that team's budget. Showback reports the same per-team/per-application cost breakdown for visibility without an actual budget transfer. Runaway agent cost risk is the specific hazard that an autonomous agent (stuck in a retry loop, given an ambiguous goal, or manipulated by adversarial input) consumes far more tokens/tool calls than any human would have authorized, with no natural stopping point the way a human operator provides.

Why It Matters: Traditional cloud chargeback models assume relatively stable, forecastable per-team consumption; agentic AI breaks that assumption because a single misbehaving agent run can spike spend by an order of magnitude in minutes, not months: this is qualitatively different from a forgotten VM instance left running. Without per-application/per-agent cost attribution and hard spend caps, showback reporting arrives too late to prevent the damage, and chargeback without real-time caps just tells a business unit after the fact that they owe an unexpectedly large bill.

What I Need to Understand:

  • Showback is the necessary first step (visibility) and should be in place before attempting chargeback, since accurate per-team attribution requires the same tracing/tagging infrastructure regardless of which model is used.
  • Runaway cost in agentic systems typically stems from: unbounded retry loops, an agent re-attempting a failing tool call without a cap, recursive sub-agent spawning, or an adversarial input designed to induce excessive tool use: each needs its own specific guard (max-retry limits, max-depth limits, hard per-task token/dollar budgets enforced at the gateway).
  • Hard budget caps enforced at the gateway or orchestration layer (kill the run when it exceeds a defined token/dollar ceiling) are a more reliable control than alerting after the fact, given how fast an agent loop can accumulate cost compared to traditional infrastructure overspend.
  • Chargeback models need to account for the higher cost variance of agentic workflows: a flat per-team allocation formula that worked for predictable SaaS licensing does not fairly represent AI spend that varies by workload complexity and failure rate.
  • Attribution granularity matters: per-application chargeback is necessary but often insufficient: for shared multi-tenant agent platforms, attribution needs to go down to the requesting user or workflow, not just the owning team.

Questions I Should Be Able to Ask My Team:

  1. Do we have hard per-task or per-session spend caps enforced at the gateway, or only after-the-fact cost alerts?
  2. What is our attribution granularity: can we trace a cost spike back to a specific agent, workflow, or user, not just a team-level total?
  3. Has a runaway-cost scenario (retry loop, recursive sub-agent spawn) ever actually occurred in our environment, and what stopped it?

Technologies / Standards / Companies to Know: FinOps Foundation, gateway-enforced budget caps (Bedrock/Azure/Vertex quota and budget features), cost-tagged tracing (LangSmith, Arize).

Recommended Learning:

Time Investment: 1 hour