Skip to main content Skip to footer

When not to reason: Routing, reuse, and deterministic alternatives for enterprise AI

Abstract

Enterprise AI architectures often invoke a language model before determining whether the task requires model reasoning. Frontier models calculate date differences, reproduce database values, answer repeated questions, apply deterministic rules, and regenerate decisions the system has already validated. The individual responses can be correct while the architecture remains inefficient, because it treats model capability as proof of model necessity.

This article defines a Model Cascade for enterprise AI and extends it into a Reasoning Escalation Cascade. The cascade selects deterministic software, authoritative retrieval, exact reuse, validated semantic reuse, smaller specialized models, frontier-model reasoning, and human judgment. Model reasoning becomes an escalation path. The default execution environment is deterministic software, retrieval, and reuse.

The article also defines the Semantic Cache Pattern, which caches a validated decision artifact together with the conditions under which it remains reusable. Reuse depends on principal, entities, policy version, evidence state, temporal validity, task contract, and risk. Embedding similarity alone is insufficient.

This article completes a set with three companions in the series, on the context, runtime, and invocation layers [1] [2] [3]. It governs the decision that precedes them all: whether a model should be invoked at all. The companion article “The intelligence contract” governs the invocation once this article’s routing decision has selected a model path [3].

Published routing studies report substantial quality-cost gains under benchmark conditions. Newer evidence also shows that several sophisticated routers fail to beat simple baselines. Semantic-cache studies reveal a similar tension: Repeated queries create material opportunity, and false hits can return the wrong answer silently. The enterprise requirement is the least expensive path that satisfies an explicit quality, latency, and risk contract. Maximum routing or cache-hit rate is the wrong objective.

Keywords: Model routing, model cascade, reasoning escalation cascade, semantic cache, deterministic software, AI inference optimization, enterprise AI architecture, reasoning reuse, AI governance

1. The model that did too much

An employee submits an IT service request:

“My VPN access stopped after I moved from Finance Operations to Corporate Audit. I need access before the 4 p.m. reporting window.”

The enterprise assistant invokes a frontier model.

The model summarizes the request, extracts the employee identifier, and calls the identity system. It reads the role-change date, checks the current VPN policy, calculates the days since the transfer, and reasons that the previous entitlement expired.

It then calls the model again to verify the calculation.

A third call reformats the recommendation as a service ticket. A fourth drafts an explanation for the employee. A fifth critiques the explanation for policy compliance.

The result is correct. Access cannot be restored automatically because the employee’s new role requires a different approval group.

Almost none of the workflow required frontier-model reasoning.

The identity system already knew the current role. A policy engine can map that role to the required approval. Standard date code can determine whether the previous entitlement had expired. A template can explain the next step. The only potentially ambiguous work was classifying an unusual free-text request and identifying the role change as the relevant cause when a network outage was the more obvious guess.

The system spent reasoning on facts, rules, calculations, and prose it already knew how to produce.

This composite scenario reflects a common production pattern. Teams begin with the model because the model makes the entire demonstration possible. They retain that architecture after the demonstration, even when task analysis shows that most steps have cheaper and more reliable implementations.

The routing decision was the point of failure. The model itself performed correctly.

2. Capability and necessity

Language models can perform arithmetic, classification, retrieval-like recall, summarization, planning, and code generation. That breadth makes them useful. It also creates an architectural trap.

When one component can attempt every step, teams stop asking whether it should perform every step.

Enterprise systems already contain components specialized for exact work. Databases retrieve authoritative values, and policy engines apply declared rules. Identity systems resolve principals and permissions, workflow engines manage state, and calculators produce repeatable arithmetic. Templates render approved communications, caches reuse prior computation, and humans decide consequential exceptions.

Model reasoning works best as a complement to those specialized systems.

Key point. Model capability establishes that a model can attempt a task. Architecture determines whether a model is the appropriate path for a given transaction.

The existing enterprise AI architecture separates memory, context, and runtime concerns [1] [2]. Memory governs what agents retain. Context governs what they see at a decision point. Runtime governs how proposed actions execute. The next control occurs before all three are allowed to trigger expensive reasoning: Should the system reason again at all?

That control is routing.

Within the Intelligence Economics narrative of this series, routing is a reasoning-allocation decision. Each tier of the cascade spends a different amount of intelligence on a transaction, and the routing policy allocates the least intelligence that will still satisfy the outcome contract. Deterministic execution, authoritative retrieval, and validated reuse raise the yield of the reasoning that remains, because they remove work that never needed a model from the paths that do. The cascade is the allocation mechanism, and precision-gated reuse and deterministic offload are the yield mechanisms.

3. From model cascade to reasoning escalation cascade

Model cascades are established research. They typically send a request to a lower-cost model first and escalate to a stronger model when the first response appears unreliable.

FrugalGPT formalized this approach across commercial model application programming interfaces (APIs). Its experiments reported that a learned cascade can match the best individual model with cost reductions as high as 98% in the best case. In separate best-case conditions, it improved accuracy by up to 4 percentage points at matched cost. Dataset-level results vary, as section 9 shows [4].

Enterprise architecture needs a broader cascade.

The first question should be which execution path should resolve this task, and only then which model should answer.

The Model Cascade in this article is therefore implemented as a Reasoning Escalation Cascade.

This evolves and expands the Model Cascade proposed earlier in the series. The original cascade routed among models, and the Reasoning Escalation Cascade routes among execution paths, of which model tiers are one subset. A reader who knows the earlier concept can treat this as the same idea widened to include deterministic execution, retrieval, reuse, and human judgment.

Pattern definition. Route each transaction to the lowest-cost, lowest-risk path that is eligible to satisfy its quality contract, and escalate only when validation, uncertainty, novelty, or consequence justifies a more capable path.

The cascade has seven potential tiers, as shown in Figure 1.

Figure 1. The reasoning escalation cascade

Source: Infosys

The cascade is an eligibility ladder. The router selects the first eligible path from task, policy, and risk signals, so a deterministic eligibility check does not first query a cache, two models, and a human. Escalation occurs only when the selected path cannot satisfy the contract.

Among the automated tiers, validated semantic reuse at tier 3 carries the highest risk of a silent error, because a false reuse returns a wrong answer without any fresh reasoning. For that reason it is introduced last in the adoption sequence in section 15, and section 6 treats it in detail.

4. The routing policy

In the terms of this series, the routing policy is where reasoning is allocated. A routing policy should optimize for more than model price.

For a request x, the system selects the path p that minimizes a weighted combination of cost, latency, and risk:

minimize over p: α · Cost (p, x) + β · Latency (p, x) + γ · Risk (p, x)

subject to a quality floor and to every eligibility constraint:

Quality (p, x) ≥ Quality_min (x), and all security, privacy, authorization, residency, and side-effect policies satisfied.

Here Cost, Latency, and Risk are the estimated cost, latency, and residual risk of resolving x on path p; α, β, and γ are weights set by the task’s service-level priorities; and Quality_min (x) is the minimum acceptable outcome quality for the task class.

The equation is a design model. It is not a requirement to compute one perfect numerical score. Its purpose is to prevent cost from becoming the only routing objective.

A cheap path that produces an unauthorized disclosure is not efficient. A fast cache hit that returns a decision under an obsolete policy is not an optimization. A highly accurate frontier response may still be inappropriate when the task contains data that cannot be sent to that deployment.

The router itself consumes resources. A classifier that calls a model on every request reintroduces an inference before the very decision meant to avoid unnecessary inference. The routing objective must therefore include the router’s own cost and latency. The design should place a deterministic prefilter ahead of any model-based classifier so that clearly deterministic or clearly cached requests never reach it. A router earns its place when its own cost sits well below the savings it produces.

4.1 Policy eligibility comes first

Before estimating difficulty, the router should determine the following:

  • Data sensitivity.
  • Tenant and principal.
  • Authorization scope.
  • Jurisdiction and residency.
  • Whether the request is read-only or state-changing.
  • Consequence and reversibility.
  • Permitted models and tools.
  • Any mandatory human-review conditions.

This step removes ineligible paths. The system should not calculate a quality-cost trade-off for a model it is prohibited from using.

4.2 Task structure comes second

The router then classifies the work along several attributes:

  • Deterministic or probabilistic.
  • Current fact or interpretation.
  • Bounded or open-ended output.
  • Repeated or novel intent.
  • Stable or volatile evidence.
  • Personalized or general answer.
  • Reversible or irreversible action.
  • Low or high consequence.

Prompt length alone is a poor proxy for difficulty. A long document may require simple extraction. A short sentence may encode a consequential legal exception.

4.3 Validation controls escalation

Every automated tier needs an acceptance test.

A deterministic path passes when typed inputs satisfy its preconditions.

A retrieval path passes when an authoritative, current, authorized source supplies the required fact.

A cached result passes only when its reuse conditions remain true.

A smaller model passes when the output satisfies the task contract and its confidence or validation signal exceeds a calibrated threshold.

A frontier model passes when evidence, semantic, and policy validation succeed.

A human path remains available when automation cannot satisfy the contract.

The router should record a reason code such as DETERMINISTIC_RULE_MATCH, AUTHORITATIVE_VALUE_FOUND, EXACT_REUSE_VALID, SEMANTIC_REUSE_VALID, SMALL_MODEL_ACCEPTED, FRONTIER_ESCALATION_NOVELTY, FRONTIER_ESCALATION_VALIDATION_FAILURE, or HUMAN_REVIEW_HIGH_IMPACT. Without a reason code, routing becomes another opaque model decision.

The same control should appear in the output contract that governs the selected path. For example, a procurement-exception contract should return the contract version, decision value, policy references, missing facts, confidence band, abstain reason, and an action type that states whether the result is recommendation-only. That prose-level contract matters more than a raw payload shape, because it tells downstream validators which claims, facts, and policy clauses must be checked before the recommendation is accepted.

5. Deterministic software is the first alternative

The most reliable model call is the one an exact function makes unnecessary.

A task is a candidate for deterministic execution when its result can be expressed as a stable function over typed inputs. Common examples are as follows:

  • Adding invoice lines.
  • Calculating elapsed time.
  • Comparing dates.
  • Validating accounts or policy identifiers.
  • Checking thresholds.
  • Applying explicit eligibility rules.
  • Converting units.
  • Sorting results.
  • Formatting a response.
  • Selecting a workflow queue from an approved mapping.
  • Enforcing output invariants.

This does not imply that natural language plays no role. A model may extract an effective date from a document or map a user’s description to a known request type. Once the inputs are typed, the calculation should move to code.

PAL tested this separation across 13 reasoning tasks. The model generated executable steps, while a Python interpreter produced the result. On GSM8K, PAL using Codex exceeded PaLM with chain-of-thought by 15 percentage points in top-1 accuracy [5].

That result does not prove that generated programs are appropriate for every enterprise workflow. It supports the underlying principle: probabilistic interpretation and deterministic execution are complementary.

5.1 Determinism needs governance too

Deterministic software can be wrong when the encoded rule is wrong or stale.

The routing layer should therefore attach the rule version, policy source, effective date, test-suite version, owner, and expiry or review date.

A hardcoded threshold with no owner gains no safety from running without a model.

6. The semantic cache pattern

Exact caches reuse results for identical keys. Enterprise language requests rarely arrive as identical strings, which is why the semantic cache pattern in Figure 2 separates semantic retrieval from deterministic equivalence validation.

“Can I carry forward unused leave?” “Does my annual leave roll into next year?” “What happens to leave I do not use by December?”

These requests may express the same intent. They may also require different answers when the employees belong to different countries, business units, contracts, or policy versions.

Embedding similarity can retrieve a candidate. It cannot establish business equivalence.

The semantic cache pattern separates those two decisions.

Pattern definition. Use semantic similarity to retrieve a candidate validated result, then reuse it only after deterministic checks confirm that the current transaction and the cached transaction are equivalent across every field capable of changing the answer.

6.1 Cache the validated decision artifact

A semantic-cache entry should contain the following:

  • Normalized task intent.
  • Task and risk class.
  • Relevant entities and typed variables.
  • Tenant and requesting-principal scope.
  • Jurisdiction or business unit.
  • Context and evidence fingerprint.
  • Policy and source versions.
  • Contract version.
  • Model and validation configuration.
  • Accepted output.
  • Supporting references.
  • Quality or review result.
  • Creation time.
  • Expiry.
  • Invalidation conditions.

The reusable asset is the validated decision and the conditions under which it remains valid; it does not depend on storing private chain-of-thought.

A semantic-cache entry is itself a sensitive data asset. It records principal scope, entities, evidence fingerprints, and policy state, so it inherits the tenant-isolation, retention, access, and residency controls that apply to the data it summarizes. It should be governed as a controlled store, held to the same standard as the data it summarizes.

6.2 Similarity proposes; equivalence disposes

Consider two requests that illustrate why Figure 2 requires a separate equivalence validator:

“Can this customer receive a refund of $450?” “Can this customer receive a refund of $4,500?”

They may be nearly identical in embedding space. The amount may cross a human-approval threshold.

Other dangerous near-neighbors are as follows:

  • The same policy question under versions 7 and 8.
  • An employee compared with a contractor.
  • Australia compared with Singapore.
  • A standard customer compared with a sanctions-screening match.
  • Read access compared with write access.
  • The same product before and after a recall.

A reuse validator should therefore check the following:

  • Principal and tenant equivalence.
  • Entity identity or declared generality.
  • Material numeric and categorical variables.
  • Jurisdiction.
  • Policy and evidence versions.
  • Data freshness.
  • Output-contract version.
  • Risk class.
  • Side-effect classification.
  • Expiry.

If any material condition differs, the result is a cache miss even when semantic similarity is high.

Figure 2. The semantic cache pattern

Source: Infosys

Note: Embedding similarity retrieves a candidate validated decision, and a deterministic equivalence validator decides whether the candidate may be reused.

6.3 Precision matters more than hit rate

A cache miss costs another computation. A false cache hit returns the wrong result without reasoning again.

This asymmetry changes the optimization objective.

A study examined more than 27,000 ChatGPT queries from 20 participants. It found that an average of 31% were similar to a previous query by the same user. That demonstrates meaningful reuse potential. The population was a consumer sample, so the figure may not transfer to an enterprise workload [6].

The same research evaluated MeanCache against GPTCache on 1,000 queries containing 700 unique and 300 repeated queries. MeanCache using MPNet achieved precision of 0.72 versus 0.52 for GPTCache and produced 89 false hits versus 233. The authors prioritized precision because a false miss can fall through to the model, while a false hit returns an incorrect cached answer [6].

For enterprise decisions, even 0.72 precision would be unacceptable in many domains. The study is useful for the metric it exposes. It does not supply a production threshold.

Key point. A semantic cache should be optimized for the validity of each reuse. Hit rate is a secondary objective.

6.4 Invalidation is part of the pattern

Time-to-live alone is insufficient.

A cache entry should invalidate on any of the following events:

  • A policy version changes.
  • An authoritative source changes.
  • The underlying entity state changes.
  • Permissions change.
  • The output contract changes.
  • A model or validator update alters acceptance behavior.
  • The business owner withdraws approval.

Event-driven invalidation is usually more reliable than waiting for a clock to expire.

7. The enterprise routing architecture

Figure 3 shows the control flow. A policy and risk gate removes ineligible paths, a classifier produces a typed routing record, candidate paths feed path-independent validators, and results are accepted or escalated to human review.

Figure 3. Routing and validation control flow

Source: Infosys

Note: A policy and risk gate removes ineligible paths, a classifier produces a typed routing record, candidate paths feed a set of path-independent validators, and results are accepted or escalated to human review.

7.1 Policy and risk gate

The gate removes paths that are prohibited by sensitivity, jurisdiction, authority, or consequence. High-impact state changes may require human approval regardless of model confidence.

7.2 Task classifier

The classifier converts an unstructured request into a routing record. That record captures the task family, entities, required data, volatility, novelty, consequence, personalization, and approved output contract.

This classifier can itself be deterministic, model-based, or hybrid. Its accuracy must be evaluated because every downstream decision depends on it.

The classifier is also a cost center. When it is model-based, it adds an inference to every request, which erodes the savings the cascade is meant to produce. A deterministic prefilter should resolve the obvious cases first so that a model-based classifier runs only on requests whose routing is genuinely uncertain. The classifier’s cost and accuracy both belong in the routing ledger.

7.3 Path executors

Each path exposes a typed interface and an acceptance test. Deterministic rules and authoritative retrieval should be invoked directly by the router, and not hidden as model tools that require the model to rediscover when to use them.

7.4 Validators

Validation should be independent of the selected path where possible. A cached answer, small-model answer, and frontier answer should all face the same evidence and business-invariant checks.

The validators differ in how they execute. Business-invariant and equivalence checks over typed fields are deterministic, and they carry most of the safety load in the cascade. Confirming that evidence supports a decision can require a model, which would reintroduce an ungoverned invocation one layer down. A model-based validator must run under its own bounded contract, and high-impact paths should prefer a deterministic check or human review over a second model confirming the first. A validator that is itself an unbounded model call is not a control.

7.5 Escalation manager

The manager records why a path failed and selects the next eligible tier. It also enforces a workflow budget, so a cascade does not become an expensive sequence that calls every model.

7.6 The human-review tier

Human review is a governed tier with its own failure profile. Treating it as a clean fallback is a mistake. Queue latency, reviewer availability, inconsistency between reviewers, and fatigue determine whether an escalation resolves within the business deadline. The tier therefore needs its own service levels, calibration, and inter-reviewer agreement measurement, held to the same standard as the automated paths. The specific staffing model, queue design, and turnaround target are institution-specific and should be set with business operations. A review tier that cannot meet the deadline of the process it protects returns the decision to the automated path under time pressure. That is the condition the cascade exists to avoid.

8. The same IT request, routed

Return to the employee whose VPN access stopped after a role change.

Under a Reasoning Escalation Cascade, the request takes a different path.

Step 1: Resolve identity and authority
The identity system resolves the employee and confirms that the requesting principal may view the entitlement state. No model is needed.

Step 2: Classify the request
A bounded classifier maps the request to ACCESS_AFTER_ROLE_CHANGE. A smaller model may perform this step when deterministic keywords and service-catalog mappings are insufficient.

Step 3: Retrieve authoritative facts
The identity system returns the old role, new role, role-change date, current VPN entitlement, and required approval group. The policy repository supplies the current role-to-entitlement mapping.

Step 4: Apply deterministic rules
A rule determines that the old entitlement must not transfer automatically. Date code confirms whether the transition grace period has expired. A workflow mapping identifies the new approver.

Step 5: Decide whether reasoning remains
If the facts and rule agree, the system has no unresolved uncertainty. The request completes without frontier reasoning.

If the role mapping is missing, two policies conflict, or the employee belongs to an unusual temporary assignment, the system escalates to a frontier model to synthesize the evidence.

Step 6: Render the response
A template tells the employee that access requires approval from the new role owner and includes the service request identifier. The runtime submits the approval workflow.

The system concentrates intelligence where uncertainty remains, without removing it from the workflow.

9. What routing research establishes, and what it leaves open

Published work demonstrates that routing can produce material savings. It also shows that routing quality is not automatic, as summarized in Figure 4.

Figure 4. Routing evidence and practitioner interpretation

Source: Infosys

No single study validates the full enterprise architecture proposed here. The evidence establishes the feasibility of its parts and the risks of evaluating them separately.

The enterprise research question is broader:

Can a governed combination of deterministic execution, authoritative retrieval, validated reuse, model routing, and human escalation reduce resource consumption while preserving accepted business outcomes?

That question requires workload-specific evaluation.

10. Five enterprise hypotheses

Hypothesis 1: A governed cascade will reduce cost at a fixed accepted-outcome threshold
FrugalGPT and RouteLLM show that model cascades can lower cost under benchmark conditions [4] [7].

Expected observation. The full cascade lowers cost per accepted outcome relative to an always-frontier baseline, without exceeding a preregistered quality-loss limit.

The denominator matters. Cost per request rewards cheap failures. Cost per accepted outcome includes retries, human correction, and rejected results.

Hypothesis 2: Deterministic offload will produce the largest reliability gain on exact tasks
PAL supports the principle that models can interpret while deterministic runtimes execute [5].

Expected observation. Arithmetic, date logic, threshold evaluation, and identifier validation show lower variance and fewer exact errors after moving to typed code.

Hypothesis 3: A workload-specific router will outperform a generic router under enterprise distribution shift
LLMRouterBench found that several sophisticated methods failed to beat a simple baseline and that a material gap remained to an oracle router [8].

Expected observation. A router calibrated on enterprise task classes, failure costs, and acceptance labels outperforms a generic difficulty router. Performance will degrade when policy, user behavior, or model versions shift unless the router is re-evaluated.

Hypothesis 4: Semantic-cache precision will matter more than cache-hit rate
MeanCache’s comparison illustrates why false hits dominate the risk profile [6].

Expected observation. A conservative cache with a lower hit rate but strict equivalence validation produces better cost-adjusted quality than an embedding-only cache with a higher hit rate.

Hypothesis 5: Frontier escalation should track novelty and consequence more closely than request length
The overthinking study shows that extended reasoning can add little value on simple tasks [9].

Expected observation. Long but structured extraction tasks remain on low-cost paths, while short, ambiguous, high-impact exceptions escalate.

11. A publication-grade routing experiment

For a practitioner, this section is reduced to one instruction. Compare any routing or reuse design against the best single model on your own workload, and measure cost per accepted outcome together with the rate of silent false reuse. The corpus, arms, and gates that follow are reference material for teams running that comparison.

11.1 Evaluation set

A first controlled enterprise study can use 6,000 historical requests. Of these, 4,000 support rule development, router training, threshold calibration, and cache construction. The remaining 2,000 are held back as an untouched final test set.

The exact size should follow workload availability and statistical power requirements. The value of this design is the separation between calibration and final evaluation. Figure 5 compares the architecture arms that the evaluation should test.

The sample should be stratified by task family, novelty, evidence volatility, personalization, risk, reversibility, and historical resolution path.

High-risk and rare cases should be oversampled in the test set, then weighted appropriately when reporting population-level results.

11.2 Counterfactual near-neighbor set

A separate challenge set should contain requests that are semantically similar but operationally different. The following pairs illustrate the risk:

  • $450 compared with $4,500.
  • Policy version 7 compared with version 8.
  • Employees compared with contractors.
  • Australia compared with Singapore.
  • Read access compared with write access.
  • Standard customer compared with high-risk customer.
  • Current status compared with status at a past date.

This set tests routers and semantic caches against the cases most likely to produce silent false reuse.

11.3 Compared architectures

Figure 5. Compared architectures for routing evaluation

Source: Infosys

Including the best single-model baseline is essential. The 2026 routing benchmark shows that advanced routers do not always outperform it.

11.4 Ground truth and review

Use deterministic validators where exact outcomes exist. For policy and judgment cases, use blinded domain reviewers who do not know which path produced the response.

Record reviewer disagreement. A task with disputed ground truth should not be treated as a clean model error.

For each response, reviewers should assess outcome correctness, evidence support, material omission, policy compliance, appropriate abstention, and suitability for the intended operation.

11.5 Measures

The core measures are defined as follows.

Reasoning avoidance rate = (transactions resolved without model inference) / (total transactions). This counts transactions closed on the deterministic, authoritative-retrieval, and exact- or semantic-reuse tiers.

Frontier escalation rate = (transactions routed to the frontier model) / (total transactions).

Semantic-cache precision = (valid semantic-cache reuses) / (total semantic-cache reuses). A reuse is valid when the equivalence validator’s conditions hold and the returned decision matches the decision the full path would have produced.

False-reuse rate = (invalid semantic-cache reuses) / (total transactions). The whole transaction population is the denominator, so that a low hit rate cannot mask the absolute rate of wrong answers.

Routing regret = the utility of the best eligible path minus the utility of the selected path identified retrospectively, where utility combines quality, cost, latency, and risk.

Additional measures are as follows:

  • Accepted-outcome rate.
  • Decision quality by task class.
  • Evidence and business-invariant validator precision and recall.
  • Validator false-accept rate on high-risk cases.
  • Exact-cache hit rate.
  • Semantic-cache hit rate.
  • Stale-reuse rate.
  • Small-model acceptance rate.
  • Human-escalation rate.
  • Human-review turnaround at 50th percentile (P50) and 95th percentile (P95), and inter-reviewer agreement.
  • Cost per accepted outcome.
  • Model calls per completed transaction.
  • P50 and P95 latency.
  • Policy-gate rejection rate.

11.6 Suggested acceptance gates

Figure 6 lists practitioner starting points for acceptance gates.

Figure 6. Suggested acceptance gates for early pilots

Source: Infosys

Note: These are interpretations and not published results.

A 99% semantic-cache precision threshold still means one wrong answer per 100 cache hits. Whether that is acceptable depends on consequence. Many financial, legal, identity, and safety workflows require a much stricter threshold or no semantic reuse.

11.7 Reporting

Report results by task family and path. A single average hides whether cost savings came from safe deterministic offload or aggressive routing of difficult cases to weaker models.

Publish the following:

  • The model and embedding versions.
  • Router training data.
  • Route thresholds.
  • Policy and contract versions.
  • Cache-key fields.
  • Cache invalidation rules.
  • Prices and measurement date.
  • Token and latency normalization.
  • Reviewer process.
  • Confidence intervals.

Routing is an empirical system. It should be evaluated as one.

12. Ownership and service levels

Figure 7 assigns ownership for the core capabilities in the cascade and identifies the minimum service-level metric each owner should maintain.

Figure 7. Ownership and service-level responsibilities

Source: Infosys

The router should not own the quality threshold it is evaluated against. Product and risk owners define acceptable outcomes. Platform engineering implements and measures the path selection.

Likewise, the team seeking a higher cache-hit rate should not unilaterally define semantic equivalence. Domain owners must identify which variables can change an answer.

13. Failure modes

13.1 Optimizing for cost alone

A cost-only router sends too many tasks to weak models or stale cached responses.

Control. Enforce quality and risk constraints before cost optimization.

13.2 Treating prompt length as difficulty

A long extraction request may be easy; a short exception may be consequential and ambiguous.

Control. Classify task structure, novelty, evidence, and impact.

13.3 Confidence without calibration

A smaller model may express high confidence on precisely the cases it does not understand.

Control. Use independently evaluated acceptance signals, deterministic validators, or calibrated classifiers. Avoid routing solely on the model’s self-reported confidence. The companion article applies the same caution to a model-supplied confidence band in the Output Contract, treating it as an input to validation and not as a trusted signal [3].

13.4 Sequential cascade latency

A weak model followed by a strong model can cost more and take longer than calling the strong model first.

Control. Predict difficulty before inference where possible. Set a maximum number of model tiers and evaluate total workflow latency.

13.5 Stale semantic reuse

A cached answer survives a policy or source change.

Control. Include policy and evidence versions in the cache entry and invalidate on source events.

13.6 Cross-principal or cross-tenant contamination

A response generated for one user or tenant is returned to another.

Control. Partition caches by tenant and entitlement scope. Keep sensitive, personalized results out of a globally shared semantic cache.

13.7 Near-neighbor collision

Two requests are linguistically similar but differ in material number, entity, jurisdiction, or action.

Control. Treat embedding similarity as candidate retrieval only. Require typed equivalence checks.

13.8 Router drift after model updates

A provider changes a model while the router retains thresholds calibrated on its previous behavior.

Control. Pin model versions where possible, run shadow evaluations, and treat model updates as routing-policy changes.

13.9 The cascade that always reaches the frontier

Poor validation or overly conservative thresholds cause every request to traverse several paths before invoking the largest model.

Control. Compare total cascade cost and latency with direct frontier invocation. Remove tiers that do not create measurable value.

13.10 Reusing the answer but not its conditions

A cache stores text without the policy, data, and authorization state that made it correct.

Control. Cache validated decision artifacts with explicit reuse conditions.

13.11 A router that costs more than it saves

A model-based classifier or a deep cascade can consume more in routing overhead than it recovers in avoided inference.

Control. Place a deterministic prefilter ahead of any model-based classifier, meter routing overhead as part of cost per accepted outcome, and remove routing stages that do not pay for themselves.

14. Regulatory alignment

The controls described in the preceding sections already produce most of what a regulated institution has to evidence. This section makes that mapping explicit. The mapping follows from the architecture and does not add a separate compliance layer. The cascade’s controls map onto obligations that apply to regulated financial institutions in the Asia-Pacific region.

Operational risk. APRA CPS 230 requires an APRA-regulated entity to identify, assess, and manage operational risks; maintain critical operations within tolerance levels through severe disruptions; and manage service-provider risks [10]. The workflow budget, the ceiling on model tiers, and the reason codes help bound and evidence the operation of the cascade. They reduce the risk that an uncontrolled sequence of model calls contributes to a breach of an institution’s operational tolerance.

Information security, residency, and access. APRA CPS 234 requires information-security controls and testing to be commensurate with the criticality and sensitivity of information assets [11]. MAS TRM guidelines provide complementary guidance on technology-risk governance, information-asset management, third-party services, security-by-design, and cyber resilience [12]. The policy and risk gate in section 4.1, the cross-tenant-reuse gate of zero, and the partitioning control in section 13.6 apply these principles before any path executes.

Fairness, accountability, and reuse. MAS FEAT principles provide foundational guidance for responsible use of artificial intelligence and data analytics in financial services [13]. SAFR is an industry-developed runtime framework, rather than binding regulatory guidance, that proposes governance checkpoints for authorizing, assessing, and recording agent actions before execution [14]. A reused decision raises a specific accountability question. The decision artifact addresses it by carrying the policy version, evidence state, and authorization scope under which the result was valid and by invalidating reuse when any of those conditions changes. Sections 6.4 and 13.5 directly address the risk of silent reuse of a stale decision.

Model risk management. Section 11 compares every automated path against the best single-model baseline and against human ground truth. That comparison is the effectiveness testing model-risk practice expects before control is relied upon. Section 11.5 measures the validators’ own precision and recall, so the control rests on measured evidence.

The trace and cache as data assets. The route trace and the semantic cache are themselves in-scope data assets. They hold principal scope, evidence fingerprints, and policy state, and they inherit the tenant-isolation, retention, access, residency, and privacy obligations of the data they summarize. An institution should govern them as controlled stores, held to the same standard as the data they summarize.

The mapping is directional. It is not a compliance opinion. An institution should confirm the obligations that apply to its jurisdiction and process class with its own risk and compliance functions.

15. Where to start

The adoption sequence should begin with the least ambiguous controls.

First, instrument the current architecture. Record task class, model, calls, tokens, tools, latency, validation, human correction, and final outcome. Without that baseline, routing becomes cost speculation.

Second, remove deterministic work from model prompts. Arithmetic, date logic, identifiers, mappings, and formatting provide the safest early gains.

Third, add authoritative lookup paths and exact caching. Both are easier to validate than semantic reuse.

Fourth, introduce model routing for bounded, low-risk task classes. Compare it with the best single-model baseline as well as the frontier model.

Fifth, add semantic caching to stable, read-only, nonpersonalized tasks. Build the counterfactual near-neighbor test set before enabling production reuse.

Finally, extend the cascade to higher-impact work only after false reuse, routing regret, policy compliance, and trace reconstruction are measurable.

Semantic caching should normally arrive later than model routing. Its best outcome avoids inference. Its worst outcome silently reuses a decision that no longer applies.

16. Reasoning should be earned by uncertainty

The enterprise creates value by resolving uncertainty reliably. The number of problems a model touches is a poor measure of that value.

Deterministic software should handle exact work. Authoritative systems should supply current facts. Exact caches should remove duplicate computation. Semantic caches should reuse only decisions whose material conditions remain equivalent. Smaller models should handle bounded language tasks. Frontier models should receive the cases whose novelty and ambiguity justify their capability. Humans should retain responsibility for decisions whose consequence exceeds the system’s approved autonomy.

That hierarchy is the Model Cascade in enterprise form.

The governing question is no longer:

Which model can answer this request?

It is:

What is the least expensive, least risky path that can satisfy this request’s outcome contract?

Once the architecture asks that question before every inference, model reasoning stops being the default implementation. It becomes a resource allocated to the uncertainty that remains.

Glossary

Abbreviation or term Meaning
AI Artificial intelligence
AIDA Artificial intelligence and data analytics
API Application programming interface
APRA Australian Prudential Regulation Authority
CPS 230 APRA Prudential Standard CPS 230 Operational Risk Management
CPS 234 APRA Prudential Standard CPS 234 Information Security
FEAT Fairness, Ethics, Accountability and Transparency
FinOps Financial operations discipline for governing technology cost and value
FrugalGPT Research approach for routing requests across language models to manage cost and quality
GPT Generative Pre-trained Transformer
GSM8K Grade School Math 8K benchmark
IT Information technology
LLM Large language model
MAS Monetary Authority of Singapore
MAS TRM Monetary Authority of Singapore Technology Risk Management Guidelines
MPNet Masked and Permuted Pre-training Network
MRM Model Risk Management
P50 / P95 50th-percentile / 95th-percentile measurement
PAL Program-Aided Language Models
PaLM Pathways Language Model
SAFR Safeguards for Agentic Finance at Runtime
TRM Technology risk management
VPN Virtual private network

References