Enterprise AI runtime architecture: The layer that turns reasoning into execution

1. The workflow that finished twice

An accounts payable agent processes a vendor invoice. The model reads the invoice, matches it to an open purchase order, and approves the payment. The reasoning is correct. The agent calls the payment application programming interface (API), and the network drops before the response returns. The agent retries exactly as its loop instructs. The vendor receives the payment twice. Finance finds the duplicate three weeks later during reconciliation, and the incident report blames the model.

The model made no errors. It produced the right decision from the information it saw. The failure happened after reasoning ended, inside the machinery that carried the decision into a business system. That machinery kept no record of the first call. Nothing deduplicated the retry, and no gate paused a money movement for review. In most enterprises, this machinery is a simple retry loop that no team formally owns. This article names the layer that should replace it.

2. From model to execution platform

LLMs generate text. Everything an enterprise experiences as agent behavior happens in code that surrounds the model: the tool calls, the workflow steps, the retries, the writes into business systems.

Anthropic’s guidance on building agents draws the boundary the same way. The model directs the process, and the surrounding system executes it [1]. The industry poured attention into the first half of that arrangement and far less into the second.

The runtime is to an AI agent what an application server is to a business application. It provides the deterministic execution services that surround probabilistic reasoning. The model decides what should happen. The runtime determines how, when, and under what controls it executes that decision.

Enterprises solved this class of problem before. Workflow engines, transaction monitors, and message queues exist because execution across unreliable networks demands architecture. Distributed systems research formalized the discipline decades ago. Sagas, for example, pair each step of a long-lived transaction with a compensating action. When a later step fails, the registered compensation reverses the earlier step’s business effect [2].

The enterprise AI runtime applies that discipline to a new kind of caller. The caller now proposes actions probabilistically, and the runtime must execute them deterministically. The runtime manages that inversion. The controls carry forward from that history; the runtime must now verify each action rather than assume it was sanctioned.

The stakes rose when agents crossed from conversation into transaction. A chatbot that answers incorrectly costs the reader time and nothing more. An agent that executes a bad decision moves money, changes records, and creates obligations. Enterprises now route agents into order management, claims processing, and customer operations, which places model output one step away from systems of record. That step is where the runtime lives, and in most enterprises it has received the least engineering attention.

3. Why the agent loop failed in production

The first generation of enterprise agents ran on one pattern. The model received a goal, picked a tool, observed the result, and picked again until it declared the task done. Frameworks made the loop easy to build, and demos made it persuasive. Production exposed its weaknesses.

The loop concentrates every execution responsibility in the least reliable component. The model decides when to retry, so a transient timeout becomes a duplicate payment. Workflow position lives in the context window, where a crash erases all progress and a long task overflows the limit. Policy exists only as prompt text, which leaves a persuasive document free to steer the agent toward actions its operator never sanctioned. The Open Worldwide Application Security Project (OWASP) names this risk excessive agency: a system granted more capability and autonomy than its task requires [3].

The execution record is a transcript, leaving auditors to reconstruct incidents from chat logs. The loop also spends without limits. An agent that retries reasoning on every failure consumes tokens at a rate no forecast anticipated, and greater autonomy multiplies the bill. Cost control belongs in the execution layer, and the basic loop offers no budget or spending limits.

Each failure traces to the same root: reasoning and execution share an undifferentiated loop. The runtime separates them.

4. What the runtime consists of

No single product delivers the runtime. It is an execution platform composed of deterministic services that surround the language model. These services receive the model’s proposed actions and execute them safely against enterprise systems.

A typical runtime combines five core services and three cross-cutting ones. A workflow engine manages orchestration: sequencing, retries, compensation, and long-running execution. A tool gateway mediates every call into enterprise systems through typed contracts, validation, and idempotency. A durable state store persists workflow progress, intermediate results, pending approvals, and execution checkpoints.

A policy engine evaluates enterprise-defined rules and returns decisions such as allow, deny, require approval, or escalate. Resilience services provide retries, circuit breakers, timeout handling, and recovery from partial failure.

Three further services carry cross-cutting concerns. Security and identity services authenticate agents, bind every action to the requesting principal, and enforce least-privilege access. Observability services record traces, logs, metrics, and execution evidence for debugging and audit. Cost governance services meter what a workflow spends across retries, repeated reasoning, and escalation. The context layer budgets the tokens a single inference sees; the runtime budgets the workflow. These services track spend per principal, per workflow class, and per destination system, and enforce budget ceilings as firmly as policy gates.

Enterprises often assemble these services from technologies they already run.

The choices vary across implementations, yet together the services form one deterministic platform around the model’s probabilistic reasoning.

5. Five capabilities the runtime must provide

A governed runtime performs five functions between the model’s proposal and the change in a business system. The core and cross-cutting services deliver these functions in combination. Security, observability, and cost governance cut across all five capabilities rather than standing as capabilities of their own. Cost per completed workflow joins the capability metrics. Figure 1 shows how the capabilities sit between model output and enterprise systems.

Figure 1. The enterprise AI runtime between model output and enterprise systems

Source: Infosys

6. Orchestration

Orchestration turns the model’s plan into a managed workflow. The model proposes steps; the runtime represents them as an explicit graph with defined transitions, timeouts, and compensation paths. The workflow lives outside the model, so long-running work survives restarts, waits on external events, and resumes after days without re-reasoning from scratch. Orchestration also bounds execution. It limits iterations, enforces resource budgets for each execution path, and ensures every workflow reaches a defined end state. Completion integrity measures orchestration quality: the share of initiated workflows that reach a designed end state instead of stalling or failing midway.

7. Tool execution

The runtime’s tool gateway provides tool execution capability. The gateway treats every model-generated action as an untrusted proposal until it validates, authorizes, executes, and records it. Every call an agent makes into a business system passes through this mediation layer.

The gateway exposes tools as typed contracts with declared inputs, outputs, side effects, and permissions. The gateway attaches an idempotency key to every state-changing call, so repeated delivery produces a single effect [4].

An idempotency key stops a technical duplicate: the retry of a call that already landed.

The key store retains idempotency keys only for the short window in which technical retries occur, typically hours rather than days. Within that window, the store holds the keys durably, so a crash does not erase the protection they provide.

The same invoice can arrive twice as separate submissions days apart, each carrying its own valid key, and each looking legitimate to the gateway. Catching that pattern is not the gateway’s role. The gateway records the business entity each action touches, such as the purchase order, the invoice, or the shipment. Policy enforcement compares new actions against the durable execution record, where those entities persist. A second payment against the same purchase order within a defined window escalates for review rather than deduplicating silently. The business application remains the authority on whether a duplicate is legitimate. The runtime detects the pattern at its chokepoint and routes the case for review.

Every call also carries the identity of its requesting principal, and the execution record preserves it. Principal identity anchors the runtime architecture. Policy gates, replay, and audit each depend on knowing whose authority an action carried. Service-to-service calls carry no end-user principal. The runtime records the full principal chain for those calls. Policy states whether thresholds bind the originating principal or the service account.

Generated code runs in sandboxes, and each agent holds the least privilege its task requires. A sandbox limits what generated code can reach; it does not confirm that the output is correct. The result returns through the gateway, which validates it against the tool’s declared contract before any downstream system receives it.

Gateway mediation provides an audit trail and a policy chokepoint; it does not guarantee that a tool’s implementation matches its declared contract. The gateway also produces execution evidence because every call, argument, and result is recorded.

Exactly-once rate measures tool execution quality: the share of state-changing calls whose effect occurred a single time.

8. State management

State management gives execution a durable home outside the model. Agent memory improves the model’s reasoning across interactions; execution state records the enterprise’s account of work in flight. The two differ in owner and in guarantee. The runtime persists each workflow’s position, intermediate results, and pending approvals in a store that survives crashes and restarts. A durable state also makes incidents reconstructable, because an investigator replays the workflow instead of rereading a transcript. Replay coverage measures state quality: the share of completed workflows whose execution history the runtime can reproduce step by step.

Replay reads the recorded history rather than repeating the work. The runtime reproduces each step from stored inputs and recorded tool results without reinvoking the model or the business system. Recorded outputs stand in for live ones, which keeps reconstruction deterministic. Replay proves the runtime behaved consistently. It does not prove the decision was sound. A workflow can replay perfectly from stale context or under a misconfigured policy. Reproducibility makes audit possible. Judging whether the decision itself was sound requires visibility across memory, context, runtime, and the business systems the action changes.

The workflow store has an internal structure. An operational tier retains a compact state for active workflows. Completed execution records move through governed hot, warm, and archival tiers according to audit, privacy, and regulatory requirements. Large documents, model outputs, and tool artifacts remain in specialized stores, and the workflow record references them rather than duplicating them. Tier transitions follow governance policy, operationalized by platform engineering, and records under regulatory hold stay in place until the hold clears.

9. Policy enforcement

Policy enforcement is the runtime’s control point over what an agent may do. The runtime classifies actions by consequence: reversible, compensable, or irreversible. Reversible actions proceed on their own. A compensable action proceeds once the runtime registers its undo. Anything irreversible above a defined threshold stops for human approval. The runtime routes the request with full context and records the approver’s decision. The runtime enforces these policies without authoring them. Risk and governance functions set the thresholds [5]. The runtime converts each threshold into a gate that holds regardless of agent reasoning. Unauthorized-action rate measures policy enforcement quality: the number of actions executed despite violating enterprise policy. The standing target is zero.

The policy specification itself carries risk. A threshold written as “escalate payments over $100,000” does not state whether a payment of exactly $100,000 escalates or executes.

Policies therefore need their own life cycle upstream of the runtime. That life cycle covers versioning, approval, and detection of drift between approved governance policies and the rules deployed for enforcement. That drift can arise at deployment, before the runtime ever evaluates the rule. The runtime enforces each rule as written and records the policy version it applied at every gate. Correctness of the rule stays with the governance function that authored it.

10. Resilience

Resilience keeps execution dependable when components misbehave. The runtime retries transient failures under the protection of idempotency keys, opens circuit breakers around degraded dependencies, and downgrades gracefully when a system disappears [6]. Observability supplies the measurement for resilience. Every action emits a trace that links the model’s decision, the policy check, the tool call, and the system response into one record. The record anchors each action to its requesting principal and preserves the policy version in force at the gate. The chain runs unbroken from the principal, through every gate that allowed or blocked the action, to the committed change.

The record is append-only. Once the runtime writes an entry, no component can alter it, including the agent that produced the action. Enforcement mechanisms vary by implementation, from cryptographic signing to write-once storage, and regulated deployments name the mechanism they rely on. Blocked actions land in the same record with the gate’s reason. Context provenance answers where information came from; the execution record answers under whose authority the agent acted. An auditor reconstructs both what happened and why the runtime allowed it.

Mean time to recovery measures resilience quality. Owners track it per workflow class and report trace coverage alongside it. Owners also pair it with cost per recovery, so an expensive recovery surfaces as clearly as a slow one. Recovery accounting also counts escalations that resolve as false alarms, so governance can tune thresholds against the escalation cost they generate.

Each capability exists somewhere in most enterprises today, scattered across agent frameworks, custom scripts, and incident-driven patches. The argument of this article is that they belong together, as one layer with owners and measurable service levels.

11. The same invoice, governed

Return to the accounts payable agent that paid the vendor twice. Under a runtime, the same decision takes a different path. Orchestration represents the payment as a workflow step with a defined timeout and a compensation path. Tool execution attaches an idempotency key, so the retry after the dropped connection lands on the completed payment and returns its result. Policy enforcement checks the amount against the approval threshold before any call fires. State management records each step, and the trace shows the timeout, the retry, and the single settlement. The vendor receives one payment.

A second submission of the same invoice takes another path. Policy enforcement matches the submission to the purchase order already settled and escalates the possible duplicate for human review. The matching rule is a governance-owned policy; governance defines what counts as the same invoice, and the runtime enforces the chosen rule. Different matching rules carry different fraud coverage and false-positive rates. The model behaves identically in both runs, and the runtime changes the outcome.

12. A reference architecture

The runtime occupies a fixed position in the enterprise AI stack. Upstream, the context layer assembles what the model sees: selected, scoped, provenance-labeled content within a managed budget. The model reasons over that window and proposes actions. Downstream, the runtime receives the proposal and owns everything that follows. Orchestration of steps, mediated calls, durable state, policy gates, and recovery paths all live in this layer. Only after orchestration, policy enforcement, mediated execution, and state recording does the runtime commit a single, auditable change to the enterprise system.

The boundary between layers stays crisp. The context layer never executes, and the runtime never decides what the model should have seen. The boundary still needs a defined feedback path. A policy gate that blocks an action returns the reason as an event. The context layer receives the event, refreshes its selection under its own policy, and the model reasons again over the corrected window. The trigger mode for that refresh is itself a policy choice: automatic re-invocation, event-driven refresh, or escalation to a human.

Automatic refresh can loop in financial workflows, so gates on money movement favor escalation. Before an irreversible action commits, the runtime also revalidates that the proposal’s context assertions remain current. Each crossing happens through a declared interface, which keeps the layers separate in practice.

Memory sits beside both. The runtime writes execution history into stores the memory architecture governs, and the context layer reads from them under its own selection policy. Together, memory, context, and runtime separate three independent enterprise concerns. Memory governs what the agent retains, context governs what the model sees, and the runtime governs what the enterprise executes.

13. Who owns the runtime

The runtime needs assigned ownership, and the assignments map to functions enterprises already staff. Platform engineering owns orchestration and state management because durable workflows are platform primitives. Application owners hold the tool contracts, since each contract encodes what a business system permits an agent to do. Platform engineering runs the gateway that enforces them. Security owns the enforcement machinery, while risk and governance functions own the policies it enforces. Site reliability engineering owns resilience and the observability pipeline that proves it. Identity and access management owns the security and identity services, binding every action to an authenticated principal under least privilege. Finance operations owns cost governance, setting the budgets that the runtime enforces.

Each owner reports its capability metric per agent, following the same discipline the enterprise already applies to service levels.

Incident reviews offer a practical check on these assignments. When a policy gate misses an action, the review should identify the owner whose telemetry recorded the gap. Many enterprises will find that no single owner holds this responsibility yet. Closing that gap turns the assignments from an organizational chart into a working control.

14. Where to start

Enterprises rarely build all five capabilities at once, and they do not need to. Tool mediation delivers the largest early return. Routing every agent call through typed contracts with idempotency keys removes duplicate side effects entirely and creates the execution record as a byproduct. Policy gates on irreversible actions come second because approval thresholds protect the business while the rest of the layer matures. Orchestration and durable state follow when workflows grow long enough to survive restarts, and resilience machinery arrives with production scale.

A team can begin with a gateway in front of its existing agents rather than a rebuilt platform. The sequencing mirrors the context layer’s adoption path: close the highest-risk gap first, then let the architecture grow around the evidence it produces.

15. What the runtime buys

Model capability advances on the vendor’s schedule, while execution discipline advances on the enterprise’s own. The runtime is the part of an AI program that the enterprise fully controls, and the investment compounds. Each governed workflow adds evidence, traces sharpen incident reviews, and tool contracts harden system boundaries over time. The disciplines are familiar because distributed systems teams have practiced them for decades.

Enterprises that pair the probabilistic caller with a deterministic execution platform can run agents inside mission-critical processes. Each action leaves evidence, and each failure surfaces while the enterprise can still stop the action.