AI token optimization: How to build cost-efficient AI

Insights

• Token waste can make otherwise viable AI use cases uneconomic at scale.
• Four complementary levers, including model routing, caching, focused retrieval, and task decomposition, attack waste at different levels of the agent stack.
• The levers compound, and a well-framed task can use less context, reuse more computation, and run on a smaller model.
• The highest-value capability is human problem framing, so organizations should treat computational thinking as a core workforce skill.

Agents built on large language models (LLMs) are moving from prototypes to production, and the dominant cost driver has become inference rather than training.

Commercial LLMs are priced per token, and costs can vary by up to two orders of magnitude, from roughly $0.02 to $30 per million input tokens. That makes architectures that route every request to frontier models such as Claude Opus 5 or GPT-5.4 difficult to sustain at scale. Uber recently exhausted its entire 2026 AI budget in four months, with power users spending as much as $2,000 per month on Claude Code. Its chief technology officer, Praveen Neppalli Naga, said the company was “back to the drawing board” on AI spending.

“Cost-per-token varies depending on the model, the context, the intelligence,” said Shruti Koparkar on the Nvidia AI Podcast. “Any tokens that are generated by a more complex model and are more interactive are going to be costlier to produce. That’s just physics. And this all works together to affect ROI on AI initiatives.”

The discipline of managing this expenditure in large businesses is called tokenomics: the study of how tokens, the unit of AI’s computation and billing, are produced, reused, and conserved across an agent.

In our work with clients, this expenditure is best managed by creating a stack of four independent but composable levers, ordered from the lowest layer of the agent architecture to the highest point of runtime control. Each is a distinct agentic thinking mode, targeting a different source of waste, and the levers work together to make AI agents much more cost efficient.

Why token optimization matters

Poor token management means that spend on AI can escalate very quickly.

Using our own agents at Infosys, we have found that an agent that routes every request to a frontier model, or reprocesses the same context on every call, or floods the LLM’s window with oversized prompts can cost two orders of magnitude more than necessary. This can be the difference between taking a use case to production or canceling before deployment.

Other facets of poor token management include over-retrieval, where too much information is used, making answers less trustworthy and driving expensive retry, correction, and human escalation loops that burn still more tokens. Generating unnecessary tokens also produces latency, reducing the responsiveness that agentic use cases depend on.

Even worse, token waste scales. With poor token discipline, cost, hallucinations, and latency all rise together as usage grows, and without careful accounting and observability, this spend can’t be tied to business value.

Of course, token management isn’t the full cost picture. There is also platform infrastructure to think about, as well as the productivity lost when AI budgets are set too tight. But poor efficiency is one of the primary reasons costs explode and overspend happens in the first place.

To help manage token efficiency and effectiveness, we suggest creating the four-lever framework, making agents more efficient across four thinking modes that create savings across the agentic stack.

The four-lever framework for AI token optimization

The four levers include adaptive thinking, reuse thinking, focused thinking, and computational thinking. Adaptive thinking decides which model runs; reuse thinking considers what computation to avoid; focused thinking determines which context enters the agent’s window, and computational thinking considers how the problem itself is framed. The first three are embedded into the agent during the design phase and operate automatically. The fourth is instituted by a human during execution and provides the greatest control over both quality and cost. Figure 1 shows the stack, ordered by increasing value and leverage.

Figure 1. The four thinking modes of AI tokenomics, with greater value and leverage higher up the stack

Source: Infosys DNA Practice

Lever 1 - Adaptive thinking, model routing, and SLMs

At the base of the stack, adaptive thinking matches each request to the least expensive model capable of handling it, rather than defaulting to a frontier model. A lightweight router classifies queries by complexity and reserves powerful models for hard ones. FrugalGPT, a model routing approach, showed that a cascade of models can match the best individual model's accuracy while cutting inference cost by up to 98%. RouteLLM, using routers trained on human preference data, reported over 85% cost reduction on a standard chat benchmark while keeping about 95% of frontier model quality.

Routing also reshapes which models belong in the agentic pool. Small language models (SLMs), those under about 10 billion parameters, are suitable for the repetitive, narrowly scoped steps that dominate agentic workflows, such as tool calls and structured extraction, where they can match larger models while running between 10 and 30 times cheaper per token. The resulting design is an agent where SLMs handle most invocations, while a frontier model is reserved for hard or open-ended reasoning.

Estimated savings: Between 40% and 98% of inference cost.

Lever 2 - Reuse thinking and caching

One level up, reuse thinking eliminates redundant computation. The primary technique is context caching (or prompt caching), which reuses recurring prompt segments such as system instructions and reference documents, so they are not reprocessed on every call. Because these are large and static relative to the variable user query, cached tokens are billed at roughly 10% of the price of fresh tokens, giving input cost reductions of between 50% and 90% for applications with heavy repeated context.

Semantic caching avoids the model call entirely by extending this idea to the response level, returning a stored answer for a semantically equivalent query. Positioned between the model and the application, context-based semantic caching reduces both cost and response time on repeated and contextual queries. For example, if an agentic assistant that answers questions over a 100-page policy manual caches the manual only once, each of the thousands of daily questions then only pays full price for the short query rather than reprocessing tens of thousands of manual tokens, and identical repeats are served from the semantic cache.

Estimated savings: Between 50% and 90% of input token cost.

Lever 3 - Focused thinking and graph-augmented retrieval

The third lever, focused thinking, governs what enters the context window, which is the maximum amount of information that the LLM can consider at one time.

Focused thinking supplies exactly the context a query needs. This depends on supplying the AI model with graphs of relationships between entities in the data, limiting over-retrieval, which increases token cost and adds noise that can trigger hallucinations, and under-retrieval, which starves the model of facts. Graph-augmented approaches index knowledge as an entity and relationship graph, where relationships between words are explicit and queryable and not buried in prose, and retrieve connected substructures rather than isolated chunks of text. Microsoft’s GraphRAG builds an entity knowledge graph and summarizes information, enabling big-picture reasoning while grounding answers in structured relationships. A metagraph-augmented approach (MAG) pushes this further for structured data, where instead of just nodes and relationships, a metagraph reasons between higher order clusters of words, and works out which ones are relevant before querying more detailed substructures.

Estimated savings: A reduction of up to 75% in token cost and fewer hallucinations, including a 50% improvement in accuracy.

Lever 4 - Computational thinking and task decomposition

At the top of the stack sits computational thinking. This is the skill of taking a business problem and posing it in such a way that the LLM can deliver the most concise and economical output.

The other three levers are engineered into the agent in advance; this one lives with the invoker, who decides at runtime how the problem is posed. In tokenomics terms, that framing sets the ceiling on how many tokens the machine must spend. A well-constructed query requires far less computation than the same problem handed over in its raw format since there’s less back and forth, ambiguity, and guesswork required by the model. To help, chain-of-thought prompting can be used to create intermediate reasoning steps, while least-to-most prompting reduces a hard problem into an ordered sequence of simpler subproblems, with accuracy rising from 34% to 74% on a hard concatenation task, like taking the last letters of a word and joining them in sequence, something that is surprisingly difficult for LLMs to do. Similarly, decomposed prompting splits the task into smaller steps that are then each routed to a specialized handler.

Estimated savings: Higher first pass accuracy that avoids retry and escalation costs, leading to a reduction of between 50% and 80% in LLM costs.

From concept to industrial practice

A large US-based industrial manufacturer operated a substantial data marketplace where identifying the right data products for a given user or agent context was a persistent challenge. Infosys deployed a metagraph-based approach (lever 3) that filtered context at runtime, delivering a 50% improvement in accuracy while lowering costs by 75%. The outcome demonstrated the value of focused thinking: less noise, more relevant context, and better business results.

Another example of the four-lever framework came from a large US-based retail chain that deployed an expert digital shelf agent to monitor how products appeared on e-commerce sites around the clock. The solution orchestrated several utility agents, one for content and compliance, and another for search positioning. Infosys implemented a model router framework (lever 1) within the AI gateway that directed each request to the appropriate model at runtime. The expert agent relied on a frontier model for complex reasoning, while specialized utility agents ran on lighter-weight models tailored to specific tasks. This allocation of capabilities reduced costs by 60%, making adaptive thinking a measurable advantage.

Best practices after deployment

Adopting these thinking modes is as much an organizational shift as a technical one. Four practices help teams capture the savings reliably.

First, treat computational thinking as a named workforce skill. Because it is the highest leverage mode and relies on humans, train architects, analysts, and product owners to decompose a business problem into a machine-optimized solution, and capture the resulting decomposition patterns as reusable playbooks.

Second, establish token FinOps, or the application of cloud financial management discipline to AI spend, and observability by capturing token count per agent, per route, and per cache layer. Budgets, alerts, and dashboards should be instituted for cost per task, cache hit rate, and frontier model share.

Third, formalize a model gateway with explicit routing and data access policy. Maintain a catalog of approved models, default to small models, escalate to frontier models only against defined quality thresholds, and govern which model classes may see which data classes.

Fourth, knowledge graphs and metagraphs should be owned, access to them controlled, and kept fresh.

Crucially, teams need not build these levers from scratch. All four thinking modes are pre-engineered and available out of the box as part of the Topaz Fabric platform, letting clients adopt adaptive, reuse, focused, and computational thinking as ready capabilities and direct their effort toward framing the right problems.

Moving ahead

Savings from this four-lever framework multiply together. A well-framed subproblem (lever 4) can be shipped over to a small model (lever 1), served largely from cache (lever 2), and grounded with a minimal context (lever 3). If routing captures 60% of cost and caching a further 70%, the combined effect creates a significant cost saving.

Because computational thinking governs everything below it, investing in the human skill of problem framing protects the returns of the automatic levers underneath. Organizations should seek out people with creativity, first-principles judgment, and expertise in prompt engineering.

Applied together, the compounding levers of this Infosys framework provide a credible route to building cost-efficient AI agents, enabling organizations to scale agentic AI without blowing the budget.