Enterprise AI cost and capacity architecture: Metering and planning

2.1 Compounding consumption

The first characteristic is compounding consumption. A customer service agent that retrieves policy documents, reasons over them, drafts a response, and self-corrects may consume 50,000 tokens for a single interaction. A batch of 10,000 such interactions per day creates meaningful spend. The cost extends beyond the model call to include embedding generation for retrieval, vector database queries, and tool execution overhead.

2.2 Path variability

The second characteristic is path variability. Two identical requests can follow different reasoning paths. One may resolve in a single model call. The other may require multiple rounds of tool use and self-correction. This variability makes cost forecasting difficult with traditional methods.

2.3 Hidden multipliers

The third characteristic is hidden multipliers. Agentic systems often call other agents or services. A supervisor agent may delegate subtasks to specialist agents, each with its own model calls and tool invocations. The total cost of the top-level request fans out across these delegations. Without tracing, the true cost remains invisible.

2.4 The three-lever cost identity

These three characteristics demand a purpose-built approach. They operate through three cost levers that form a compact identity: Total cost equals the sum, across all calls in a task, of context size multiplied by effective per-token price, plus output cost.

The first lever is per-call context size: the total tokens in a single request, including the prompt, tool schemas, conversation history, and observations. The second lever is the number of large language model (LLM) calls required to complete a unit of work. Each additional call multiplies the first lever because the growing context retransmits on every turn. The third lever is the effective per-token price after cache discounts and model routing.

Naive optimizations often shift cost between these levers rather than removing it. A 40% reduction in per-call context that triggers 67% more calls reaches break-even under the linear cost model (0.60 multiplied by 1.67 equals approximately 1.00). The interaction compounds further when context grows turn over turn, because each added call retransmits accumulated history rather than a fixed-size payload. Evaluating any optimization against all three levers simultaneously guards against this substitution effect.

2.5 Workload archetypes and platform tiers

Input tokens dominate spend in tool-heavy coding agents. Cost analysis of coding-agent workloads, typified by systems such as SWE-agent [2], shows approximately a 100:1 input-to-output ratio, with roughly 99% of tokens serving as input [3]. This ratio has a practical consequence: output-side optimizations such as concise-answer instructions or token-efficient serialization move approximately 1% of the total bill. The levers that matter for input-dominated agents are prompt caching, tool-definition compression, observation management, and call-count reduction.

Reasoning-model workloads and content-generation agents sit at materially different ratios, often 3:1 to 10:1 input-to-output. Extended thinking traces and long-form generation shift spend to the output side. For these workloads, generation budget control and reasoning-token caps remain first-order cost drivers and deserve first-class treatment alongside the three input-side levers. Organizations profile each agent class by its input-to-output ratio and apply optimization effort proportionally.

Enterprise AI consumption also varies by platform tier. Embedded copilots and plugins, such as coding assistants and productivity copilots, follow per-unit or fixed-token pricing where usage visibility depends on vendor dashboards. Enterprise software-as-a-service (SaaS) platforms, such as SAP Joule, Salesforce Agentforce, and ServiceNow AI Agents, use per-action consumption models where the three-lever cost identity does not apply directly. For these platforms, the governance layer relies on vendor-provided consumption reporting and procurement-level cost controls rather than token-level metering.

Hyperscaler platforms, such as Azure OpenAI and AWS Bedrock, expose hidden costs across cloud-native services including safety classifiers, serverless knowledge bases, and agent runtime consumption. Enterprise agentic platforms provide the most direct cost transparency through AI gateways and private endpoints. Observability and control requirements increase as organizations move across these tiers.

5.1 Throughput budgeting and capacity arithmetic

Capacity planning for agentic workloads requires explicit throughput budgeting in tokens per minute (TPM) and requests per minute (RPM). Provider rate limits impose hard ceilings that spending alone cannot lift on short notice. A capacity plan allocates TPM and RPM across tenants, agent classes, and priority tiers, with reserved quotas for critical workloads and burst allowances for spikes.

A worked example illustrates the arithmetic. Consider a deployment of 50 concurrent agents, each consuming an average of 30,000 input tokens and 2,000 output tokens per turn, with an average of 4 turns per task. Assuming an average end-to-end turn latency of 60 seconds, peak concurrent turns reach roughly 50 per minute. The TPM requirement is approximately 50 multiplied by 32,000, yielding 1.6 million TPM. Against a 2 million TPM ceiling, this leaves 20% headroom for burst absorption. At 30% burst headroom the deployment fits comfortably. Below 15% headroom, a single spike from long-context trajectories can starve other tenants and trigger queuing delays. In production, capacity plans size against the 99th-percentile (P99) turn latency rather than the mean because long-tail trajectories drive the peak throughput requirement. This article uses the notation P99 to mean the 99th percentile, P50 for the 50th percentile, and so on. A P99 value is the threshold below which 99% of observations fall, making it a practical measure of worst-case behavior.

The following discussion quantifies the path-variability characteristic identified in Section 2.2 as a cost distribution rather than a mean. The cost distribution across agent trajectories matters more than the average for both budgeting and service-level agreement (SLA) design. The 50th-percentile (P50) cost represents the modal call cost that most tasks incur. The P99 cost represents the budget ceiling that capacity plans accommodate. The P99-to-P50 ratio indicates workload spikiness. A ratio above approximately 8 indicates that a small fraction of trajectories carry most of the spend. This pattern rewards targeted trajectory-shaping rather than average-case optimization. Cost-outcome reports that quote only averages systematically understate the risk from these long-tail trajectories.

Failover routing interacts with capacity arithmetic in two ways. First, a failover target has sufficient reserved TPM to absorb redirected traffic without itself saturating. Second, the failover model may carry different pricing and different context-window limits, which changes both the cost identity and the effective capacity per request. Capacity plans model primary and failover paths as a combined throughput envelope rather than treating failover as an exception.

5.2 Context optimization

Prompt caching adds a fourth factor to capacity planning. Foundation model providers offer mechanisms to cache repeated context prefixes so that organizations pay a reduced rate for cached input tokens. Depending on the provider, savings can reach up to 90% on the cached portion [5]. Realized cache hit rates often fall well below projections.

A single changed character inside the cached prefix, such as a timestamp, a reordered JavaScript Object Notation key, or a per-user variable, invalidates everything after it. Capacity planners model the effective cacheable fraction of each workload rather than assume theoretical savings. Stabilizing and enlarging the cacheable prefix moves the effective cache hit rate from single digits toward 70% or higher. The technique is direct: order static content first and push volatile content to the tail. Capacity planners also account for provider context-window pricing tiers. Several providers charge a premium above a token threshold, such as long-context surcharges above 200,000 tokens. These tiers create nonlinear cost curves: a workload sitting just above a tier boundary can be materially cheaper when compacted below it. Context optimization therefore targets two goals simultaneously: maximizing cache hit rates and staying below the pricing tier boundary that triggers premium rates.

Context management strategy also depends on whether the agent workload is read-heavy or write-heavy. Read-heavy agents, such as those performing research, debugging, or retrieval-augmented generation, consume large volumes of external content that the agent did not author. This content can be safely masked or compacted once it is no longer needed verbatim. Evidence from software engineering benchmarks shows that masking old observations can halve cost while maintaining or exceeding the task success rate.

Write-heavy agents, such as those generating code or authoring documents, produce content that they may need to reread later. Compacting this authored content triggers expensive rewrites because the agent interprets truncation as corruption of its own work. Capacity planners classify each agent by this read-write profile and apply compaction strategies accordingly.

Precision context engineering offers a complementary approach that operates at the retrieval stage rather than after ingestion. Instead of loading entire documents into the context window and compacting them later, precision retrieval identifies and injects only the exact paragraphs needed to answer a given question. This approach resembles a research assistant who hands the agent the relevant page rather than the whole library.

Precision context engineering applies the context-size lever of the three-lever cost identity at the retrieval stage. It can substantially reduce input tokens while simultaneously improving answer accuracy and reducing hallucinations. It is especially effective for enterprise knowledge base and wiki question-answering, large codebase search, natural language to Structured Query Language (SQL) translation, and document analysis and summarization. The key architectural requirement is a retrieval layer that can perform fine-grained, paragraph-level indexing rather than document-level retrieval.

6.1 Model selection and routing

Not every agent task requires a frontier model. A simple classification step may perform equally well on a smaller, less expensive model. An agentic task requiring complex reasoning and tool orchestration may justify the cost of a frontier model. Model routing exploits these differences to manage cost without sacrificing quality.

A model routing architecture evaluates each task against a set of criteria. These criteria typically include task complexity, latency requirements, accuracy thresholds, and cost constraints. The routing engine then selects the most cost-effective model that meets all criteria. Data residency and sovereignty requirements act as hard constraints on routing: workloads subject to jurisdictional rules may be ineligible for lower-cost regions regardless of the per-token saving.

Organizations commonly implement three tiers. The first tier uses small, specialized models for high-volume, low-complexity tasks such as classification, extraction, and simple formatting. The second tier uses midrange models for moderate-complexity tasks that require some reasoning. The third tier reserves frontier models for complex reasoning, multistep planning, and tasks where accuracy is critical.

The key challenge is determining which tier a given task requires. Static rules work for well-understood task types. Dynamic routing uses lightweight classifiers or heuristics to evaluate task complexity at runtime. A practical starting point assigns tiers based on task category and promotes individual requests to higher tiers upon failure or low-confidence outputs. The router itself is a cost item. A routing classifier that costs 300 tokens per decision and saves 800 tokens on average by selecting a smaller model breaks even at roughly 40% small-model routes. Semantic routing is net-positive for workloads with a clear bimodal complexity distribution, where a substantial fraction of requests can be handled by a cheaper tier. For uniformly complex workloads, the router pays its cost without producing meaningful savings.

Model routing also creates a natural hedge against provider pricing changes. Organizations that depend on a single model from a single provider carry concentrated cost risk. A routing architecture that distributes workloads across models and providers reduces this concentration. Multiprovider routing creates tension with prompt-cache warmth. Prompt caches are provider-specific, model-specific, and typically session-scoped or short-window-scoped. Routing the same logical workload across multiple providers fragments cache pools and collapses the realized hit rate. Two resolutions address this tension. The first routes at the workload-class boundary rather than the request boundary, so cache pools stay coherent within a workload. The second accepts lower cache hit rates as the price of provider diversification and factors that cost into the routing economics.

Semantic routing extends static tiering by using intent and complexity signals to drive model selection. Rather than relying solely on task category labels, semantic routing examines the request itself, and routes based on the reasoning demand it implies. This approach can achieve 40% to 85% variable cost reduction compared to routing all traffic to a frontier model [6].

6.2 Consumption pricing versus committed capacity

The cost identity behaves differently under committed-capacity pricing. Azure OpenAI provisioned throughput units (PTUs), AWS Bedrock provisioned throughput, and negotiated committed-spend agreements with foundation model providers convert the cost model from marginal (per-token) to average (per-unit-of-committed-capacity). This inversion changes the routing objective. Under consumption pricing, the routing engine minimizes per-call price. Under committed capacity, the routing engine maximizes utilization of the committed pool and spills to on-demand only when the pool saturates. Cache warmth and batching economics also shift: A committed pool rewards concentrating traffic to fill capacity, whereas consumption pricing rewards distributing traffic to exploit cache and batch discounts.

Organizations model both pricing regimes against their metering data before committing. The break-even point depends on sustained utilization. Below roughly 60% sustained utilization, committed capacity often costs more than on-demand consumption. Above that threshold, the economics tilt toward committed capacity, especially for high-volume, predictable workloads. The four-layer architecture accommodates both regimes. The metering layer captures utilization against committed pools. The planning layer models break-even. The control layer routes to fill committed capacity first. The governance layer reports effective unit cost across both pools.

6.3 Complementary cost levers

Three additional cost levers complement model routing.

The first is fine-tuning. For tasks that a business runs repeatedly with consistent context and data, a fine-tuned smaller model can deliver comparable quality at 10 to 100 times lower per-inference cost [7]. The trade-off is the total cost of ownership. Fine-tuning exchanges variable API spend for fixed training costs, ongoing retuning as base models improve, evaluation harness upkeep, and drift monitoring. The per-inference saving deserves weighing against this full operational burden. An additional risk is base-model depreciation. A model fine-tuned on a current-generation base is frequently outperformed by the next generation’s off-the-shelf base within 9 to 12 months. At that point, the fine-tuning investment is reset. Fine-tuning is a sound optimization when the task, data distribution, and quality bar are stable. It carries higher risk when the base-model frontier is moving rapidly.

Fine-tuning is most effective for industry-specific and domain-specific tasks: structured field extraction from invoices, contracts, tickets, and logs, content classification, intent detection and routing, and code migration between specific language pairs.

The second is semantic response caching. When different users ask semantically equivalent questions, a cache that recognizes intent rather than exact string matches can serve the same answer without invoking the model. This approach can eliminate 40% to 60% of calls for high-repetition workloads such as code queries and frequently asked questions [8]. Semantic response caching and prompt caching compose rather than substitute. Semantic caching sits above the model call and can eliminate it entirely.

Prompt caching sits below the model call and reduces the cost of calls that do go through. The correct architecture applies to them as a two-stage funnel. Semantic cache comes first to skip the call. Prompt cache comes second to reduce the cost of calls that proceed. The routing and generation-control levers follow. Semantic caching requires careful correctness guardrails. Semantically similar questions frequently have context-dependent answers that vary by tenant, entitlement, time sensitivity, or personalization. Cache keys scope to these dimensions, and an invalidation strategy paired with a correctness-monitoring loop that samples cache hits against fresh generations accompanies any deployment.

The third is asynchronous batch processing. Nonurgent tasks can be queued for scheduled batch execution at discounted API pricing [5]. Examples include offline codebase documentation generation, bulk content localization and translation, and repository-wide code security and vulnerability scanning. Batch processing typically achieves 50% savings on eligible work. The output quality is identical to real-time processing.

6.4 Tool design and self-hosting

Tool design also functions as a cost lever that compounds across every call in a session. Tool schemas and definitions load on every LLM invocation. Compressing verbose docstrings, merging overlapping tools, and deferring rarely used tools through lazy loading directly reduce per-call token overhead. Lazy loading of tool definitions has demonstrated reductions of up to 85% in tool-definition token usage [9]. Capping tool output size prevents outlier observations from bloating context, provided the cap includes guidance on how the agent can retrieve additional detail when needed.

For organizations operating at sufficient scale, strategic self-hosting extends the tiering model beyond API-based providers. The approach uses telemetry to identify the highest-volume, narrowest tasks and then moves those tasks to open-source or fine-tuned models running on the organization’s own cloud infrastructure. Embedding generation for retrieval-augmented generation pipelines, internal text classification and routing, and sovereign AI use cases are typical candidates.

Self-hosting converts unpredictable, variable API costs into fixed, controllable infrastructure expenses. At scale, this conversion can reduce costs by 50% or more on eligible workloads, but the realized saving depends on graphics processing unit (GPU) utilization. On dedicated GPU inference, cost per successful task is a function of sustained utilization. Below roughly 30% realized utilization, self-hosting is frequently more expensive than the API alternative. Above 60% to 70% utilization, the economics tilt sharply toward self-hosting. Batching strategy, request queuing to raise utilization, and dynamic GPU scaling patterns are the operational levers that determine where a workload lands on this curve. The trade-off also includes the operational burden of model serving, scaling, and maintenance. Organizations begin self-hosting only after telemetry confirms which workloads have sufficient volume and narrow enough scope to justify the infrastructure investment.