Enterprise AI cost and capacity architecture: Control and governance

2.1 Budget framework and circuit breakers

A budget framework operates at multiple levels and is defined in two dimensions: currency and throughput. Organization-level budgets set total spend ceilings, tokens-per-minute, and requests-per-minute allocations for a billing period. Team-level budgets allocate portions of the organization budget to specific groups. Agent-level budgets constrain individual agent types or instances. Transaction-level budgets cap the maximum cost of a single business transaction.

Circuit breakers complement budgets by detecting and interrupting runaway consumption in real time. An agent caught in a reasoning loop may generate thousands of tokens per second without producing useful output. A circuit breaker detects this pattern, typically through token-rate thresholds or iteration-count limits, and terminates the execution.

Graceful degradation policies define what the system should do when a budget runs out or a circuit breaker trips. The degradation ladder sequences actions by cost: route to a cheaper model first, serve a cached response second, defer to batched processing third, and escalate to a human operator last. Human escalation typically costs two to three orders of magnitude more per resolution than even the most expensive frontier-model call, so it belongs at the end of the ladder. For customer-facing workloads, cost policies and service-level policies require codesign. A circuit breaker that triggers mid-transaction is a service-level event, and in financial services it may also be a regulatory event. The degradation path for these workloads preserves the service-level contract through a model downgrade, cached answer, or holding response with human follow-up, rather than an outright termination.

These controls require careful calibration. Overly aggressive limits disrupt legitimate workloads. Overly permissive limits fail to prevent waste. Organizations start with conservative limits and adjust them based on observed consumption patterns from the metering layer.

2.2 Adversarial cost inflation

The control layer addresses deliberate cost inflation alongside benign runaways. A determined actor can drive spend through several vectors. These include prompt injection that triggers expensive tool chains, retrieval poisoning that inflates context on every query, and adversarial inputs that push a reasoning model into its maximum thinking budget. Unauthenticated public endpoints that let external traffic drain a quota represent a further vector. This last vector, sometimes called denial-of-wallet, is the cost-side analog of a denial-of-service attack.

Three containment mechanisms address these vectors. First, per-principal cost quotas limit the spend attributable to any single user, application programming interface (API) key, or external session within a rolling window. Second, anomaly detection on cost trajectories catches gradual inflation that falls below absolute thresholds but deviates from the principal’s historical pattern. Third, hardening of tool-call surfaces against injection-driven inflation constrains which tools an externally triggered request can invoke and caps the token budget for any single tool-result observation.

2.3 Call-count reduction

Call-count reduction deserves particular attention because it delivers the highest return on investment among all cost-optimization categories. Each additional large language model (LLM) turn resends the growing context, so cutting turns saves cost quadratically. Three techniques are especially effective.

First, actionable error messages: ambiguous tool feedback such as “more results may be available” causes agents to loop, while clear terminal states and specific error descriptions stop the loop. This single change often produces the largest cost reduction.

Second, per-tool retry circuit breakers: retry storms multiply across layers, so three retries across five layers can produce 243 calls. The circuit breaker counter operates per tool, not globally, and never retries errors that should not be retried, such as a hallucinated tool name. Third, parallel tool batching: encouraging agents to issue independent tool calls in a single turn reduces total turns while improving accuracy.

A structural approach to call-count reduction uses state machines and reusable skills to constrain agent execution paths. Agents left to reason freely can spiral into dozens of expensive API calls as they explore, backtrack, and retry. State machines impose a fixed, predictable sequence of steps with clear exit points. This converts open-ended brainstorming into an assembly-line execution pattern that eliminates runaway loops and caps operational spend.

The approach is especially effective for agentic process design, task breakdown, classification and routing, data extraction pipelines, and document transformation workflows.

A complementary lever equips agents with reusable skills: codified, prevalidated procedures for repeatable tasks. The agent identifies an available skill or authors one on first encounter, rather than rederiving the approach on every run. This collapses the plan-and-explore phase and improves consistency. As the skill library grows, the agent rederives less and cost per task trends downward. Together, state machines and reusable skills can reduce call counts by several times compared to unconstrained agent execution.

2.4 Generation and reasoning budget control

Generation and reasoning budget control addresses cost on the output side of the equation. Output tokens typically cost several times more than input tokens on major providers such as Anthropic [2], and on reasoning models the thinking-token budget can dominate spend altogether. Extended thinking budgets can inflate spending by 5 to 10 times the visible output cost, depending on the provider, model configuration, and task class. On some providers, the API response does not surface reasoning-token usage, which remains invisible to teams instrumenting only for completion tokens. On providers that do return reasoning content, capturing it requires deliberate telemetry choices. Organizations instrument reasoning-token counts as a first-class telemetry field alongside input and output tokens. The decision of when to enable extended thinking versus routing to a nonreasoning model of comparable quality on the task class is itself a routing decision. It belongs in the planning layer. Five techniques manage this cost.

First, cap maximum output length based on the task type. A classification task does not need a paragraph of explanation. Second, use constrained or structured decoding to force concise, schema-valid output rather than allowing the model to generate conversational filler. Third, strip unnecessary preamble and conversational framing from agent outputs. Fourth, tune the reasoning or thinking budget to match task complexity. A simple extraction task does not warrant a deep reasoning chain. Fifth, bound the thinking budget per task tier and route tasks that do not benefit from extended reasoning to a nonreasoning model. This routing decision applies to the effective-price lever from the three-lever identity: the same task completed without a thinking budget carries a fraction of the per-token cost.

These controls can meaningfully reduce output and reasoning token costs. They are most impactful for reasoning-model workloads, structured data and field extraction, intent detection and routing, and high-volume calls where output length would otherwise be unbounded.

2.5 Antipatterns

Several common practices that appear to reduce cost shift it to other levers or produce net negative results. Compacting content that the agent authored triggers rewrites that cost more than the original generation. Trusting theoretical cache hit rates without measuring realized cached tokens per stream leads to overestimated savings. Verbose behavioral instructions, such as spelling out parallel-batching rules at length, can cause the model to behave more sequentially rather than less. Injecting mutable state into every tool response breaks parallel batching.

These antipatterns reinforce the cardinal rule of the three-lever identity: total cost equals the sum, across all calls in a task, of context size multiplied by effective per-token price, plus output cost. Optimizing one lever in isolation risks inflating another. Every proposed optimization deserves evaluation against all three levers simultaneously, using the metering layer’s per-component attribution data, before deployment.

4.1 Maturity staging

Organizations at different maturity levels sequence their investments accordingly.

Organizations beginning agentic AI adoption prioritize metering. Instrumentation that captures per-call and per-transaction cost data creates the foundation for every subsequent capability. Without this data, routing decisions, budget policies, and governance reports all rely on estimates.

Organizations with established agent deployments focus on model routing and budgets. These capabilities produce the most immediate cost impact. Routing tasks to appropriate model tiers typically reduces cost by 40% to 60% for workloads that currently use a single frontier model. Ong et al. [4] report cost savings of over 50% to 75% on public benchmarks, with realized savings varying by workload type and complexity distribution.

Organizations operating at scale invest in cost governance and closed-loop optimization. Chargeback models, cost-outcome reporting, and dynamic routing based on real-time budget consumption represent the mature state of cost and capacity architecture.

Within each maturity stage, a specific sequencing of optimizations maximizes impact while minimizing risk. The first step at any stage is a token attribution audit: profiling per-component token counts across representative sessions to build a Pareto chart of where tokens and costs concentrate. The second step is prompt caching: reordering context so static content comes first, stabilizing the prefix byte-for-byte, and using long time-to-live settings for shared prefixes. This step carries the lowest risk and requires zero behavior changes. The third step is tool-definition compression and lazy loading. These three steps form the foundation.

Subsequent steps depend on agent type. For read-heavy agents, observation masking beyond a recency window can cut costs by roughly 50% at no quality loss. For write-heavy agents, the focus shifts to external storage pointers for authored artifacts and call-count reduction through actionable errors and retry circuit breakers. Cross-turn summarization, triggered at 70% to 80% context utilization, is deferred until masking alone no longer frees sufficient space.

4.2 Measurement discipline

Every optimization requires disciplined measurement before rollout. Agent runs exhibit high variance, so single-run comparisons are unreliable. Organizations maintain a fixed, replayable evaluation set of representative tasks: roughly 30 cases for gating individual changes and roughly 200 for regression testing across model or architecture changes.

Teams run each variant multiple times because the gap between single-pass and multi-pass reliability can be substantial. Kapoor et al. [5] demonstrate that single-run success rates are misleading and that agent accuracy varies significantly across runs, even under identical conditions. Simple baselines can match or exceed complex agent architectures once cost is controlled for, a finding visible only through repeated evaluation. Reports include multi-trial reliability rates and variance alongside single-run averages. Any optimization holds task success rate within a defined threshold, such as no more than a 3% drop, before rollout. At deployment, organizations canary each change on a sample of live traffic with automatic rollback triggered by success-rate drops or error-metric spikes. The cheapest agent that fails its task is infinitely expensive, so quality remains the binding constraint on cost optimization [5].

4.3 Minimum-viable pattern for subscale teams

The cost architecture itself carries cost. The metering layer requires observability infrastructure and instrumentation effort. The planning layer requires evaluation compute and routing infrastructure. The control layer requires gateway infrastructure and policy engines. The governance layer requires FinOps operations and reporting. Reports from production deployments typically place these overheads in the range of 5% to 15% of the AI spend they govern. The realized figure varies by deployment complexity and existing infrastructure maturity.

Below a certain scale, architecture costs more than it saves. Organizations below that threshold implement a minimum-viable cost architecture: per-call metering with a correlation identifier, a single model tier with a spend ceiling, and a monthly cost review. This starting point captures the data needed to justify further investment when scale warrants it.

Regardless of maturity, one principle applies universally. Cost and capacity management is an architectural discipline that requires the same design rigor, operational investment, and continuous improvement as any other enterprise architecture concern.