Skip to main content Skip to footer
Frontier Tech Perspectives

Enterprise AI evaluation architecture: From a testing phase to a permanent layer

Uday Kumar Gupta Uday Kumar Gupta

Raktim Singh Raktim Singh

Enterprise AI agents reason probabilistically, choose their own execution paths, and can drift without a formal change event. Conventional verification rested on three assumptions: variance was removable, a design-time reference existed to test against, and the measuring instrument stood outside the system it measured. Agents remove all three.

As a result, testing moves from a release phase to a permanent architectural layer. This article presents a reference architecture for that layer, identifying six recurring evaluation problems, tracing each to its conventional precedent, and mapping each to the capabilities that answer it.

The architecture comprises seven capabilities across four planes: data, scoring, decision, and governance. Privacy controls and cost governance apply across all seven as cross-cutting concerns. Evaluation datasets sit at the center because every downstream measure inherits their quality. Dataset design, sourcing, versioning, and contamination defense therefore become core engineering work. Scoring rubrics and judge calibration make model-based grading reproducible. Statistical regression methods replace deterministic diffing for nondeterministic systems, and continuous evaluation converts production traces into standing evidence. That evidence supports graded autonomy. Each permission level carries a rising evidence bar and a recorded grant binding the decision to the dataset version, rubric version, scores, and approver. Evaluation findings then route back into the memory, context engineering, and runtime layers this series describes.

Keywords: Enterprise AI evaluation, agent evaluation, evaluation architecture, evaluation datasets, model-based judging, autonomy governance, regression testing, continuous evaluation

1. What weak evaluation costs in production

Agentic systems changed what enterprises must verify. Traditional software runs the code that developers write, and the same input produces the same outcome. Agents introduce probabilistic reasoning, dynamic retrieval, tool orchestration, memory, and multistep trajectories. The same request can legitimately produce several correct responses through different execution paths. That premise is now widely accepted.

The next question is what an evaluation capability should contain. A composite scenario makes the answer concrete. No single incident inspired it. Each element draws on failure patterns that enterprises have seen repeatedly across industries.

An insurer deploys an agent to answer claim status questions and guide customers through appeals. Before launch, the team runs a suite of 300 prompts, and every prompt returns a correct answer. The release gate passes, and the agent goes live with authority to state policy without review.

Three months later, a customer asks whether a denied claim can still be appealed. The policy sets a 45-day appeal window, and the denial happened 46 days earlier. The agent skips the policy lookup, infers a 60-day window from a related product, and confirms the appeal remains open. The customer files, the appeal fails on the deadline, and the complaint reaches the regulator with the transcript attached.

The postmortem widens the picture. The launch suite runs once per prompt, and reruns now return different verdicts on identical inputs. The model behind the agent receives two upgrades after launch, and no gate reruns. The team has pasted 12 suite prompts into the prompt library as few-shot examples, so the agent has partly seen its own exam. The regulator asks who approved unreviewed policy answers. The scores sit in a dashboard, the approval sits in meeting minutes, and no record connects the two.

Elements of this scenario appear in documented agent failures. The organization could state what its system should do. It held no current evidence about what the system did across the inputs it met in production. Producing that evidence, and refreshing it, is the work of the evaluation layer. Six recurring problems, named in the next section, define what that layer must overcome.

2. Six problem statements

Enterprises building an evaluation capability run into the same six problems. None of them arrived with agents. Each has a conventional precedent, and each stayed manageable because a control rested on an assumption that agents remove. Naming the problem, the precedent, and the broken assumption makes the architecture that follows testable because each capability has to answer at least one of them. Figure 1 summarizes the six problems, the precedent each one carries, and the capabilities that answer it. Section 3 defines those capabilities.

Figure 1. Six evaluation problems

Source: Infosys

2.1 Unstable verdicts — the same input scores differently across runs

The same prompt yields different completions across runs, even at conservative settings. The precedent is the flaky test, which conventional teams treat as a defect and repair until determinism returns. Sampling variance offers no such repair because variance is how the model works. Measurement shifts from confirming an answer to characterizing a distribution, and a single passing run carries almost no information. In production, the same suite passes automated checks but misses failures that reach users, or swings between pass and fail on an unchanged system. Confidence in the suite erodes, and teams stop gating releases on it.

2.2 Failures inside the trajectory — a reasonable path fails several steps later

Agents work through sequences of reasoning, tool calls, and observations, and a reasonable early choice can commit the agent to a path that fails several steps later. The precedent is the multistep workflow failure in business process and distributed systems. Those disciplines can test the path because the process graph exists at design time. Agents generate the path at runtime, so no reference topology exists, and a trajectory can go wrong while every step looks locally reasonable. The symptom is a correct-looking answer reached through wrong tools, skipped approvals, or unsafe steps. The cost is policy breaches that output-level checks can miss.

2.3 No single reference answer — many valid outputs resist a fixed key

Agents satisfy the same request in several valid ways, so verification shifts from matching an answer to judging one. Descriptive outputs posed this problem long before agents, in search relevance, translation, and document drafting. Conventional programs hold it at the edges and hand it to human reviewers whose competence the organization takes on trust. Agents move judgment to the center of verification and hand much of it to a model. The judge becomes an engineering artifact with its own versions, drift, and failure modes. In practice, reviewers disagree on pass and fail for the same response. Standards drift across teams, releases, and audit periods.

2.4 Leaking, aging test data — the suite leaks and drifts from production

Teams lack cases that represent real workflows, and they lack coverage of rare and adversarial inputs. The precedent is benchmark leakage in machine learning, where contamination happens at training time inside a pipeline the team controls [1]. Agent systems open leak paths at runtime, through prompt libraries, retrieval indexes, memory writes, and model providers that trained on public material. The measuring instrument no longer stands outside the system it measures, so the dataset needs active defense. Once a suite leaks into the system it measures, its scores no longer carry information. The symptom is high scores that production experience contradicts within weeks. The cost is autonomy granted on evidence that measures nothing. Section 5 treats this problem as the core of the architecture.

2.5 Certification decay — a clean release separates from the running system

A release gate certifies one configuration on one day, and nothing more. The precedent is configuration drift, which change management contains because every backend change raises a ticket, and the ticket triggers retesting. Agent systems drift without a change event. Model providers upgrade silently, data distributions shift, and users find new phrasings, so nothing signals that the certified system and the running system have separated. Quality complaints surface months after a clean release, with no code change to blame. Incidents then become the detection mechanism.

2.6 Evidence detached from authority — scores and permission decisions never meet

Scores accumulate in dashboards while permission decisions happen in meetings. The precedent is the requirements traceability gap that audit disciplines have managed for decades. Those disciplines bind binary evidence to a one-time release of an artifact that can no longer change. Agent evidence is statistical, the decision is a standing level of autonomy, and certification decay erodes the evidence while the authorization stays in force.

The enterprise needs a record that binds each decision to the dataset version, rubric version, scores, and approver. It must refresh that binding on a cadence.

This surfaces as dashboards with no owner, no threshold, and no gate. The enterprise may not be able to reconstruct its autonomy decisions for an auditor.

Three assumptions connect the six. Conventional controls assumed that variance was removable, that a design-time reference existed to test against, and that evaluation stood outside the system it measured. Agents remove all three, and the architecture in the next section replaces what those assumptions once provided.

3. The evaluation reference architecture

The evaluation layer is a permanent part of the enterprise AI stack, alongside memory, context engineering, and runtime. Figure 2 arranges its seven capabilities into four planes. The planes separate concerns that enterprises often mix: holding evidence, producing scores, deciding on those scores, and recording the decision. The seven capabilities are evaluation datasets, evaluation store, rule-based checks, model-based judges, human review, regression harness, and continuous monitoring. Four supporting elements complete the layer but are not capabilities of their own: the trace pipeline, which serves the capabilities as transport; scoring rubrics, which are versioned artifacts the judges apply; and autonomy grant records and governance reporting, which are the outputs the governance plane produces.

Figure 2. The enterprise evaluation architecture reference model

Source: Infosys

The data plane holds what evaluation runs against. Evaluation datasets supply the cases, the fixtures they need, and the risk weighting that decides how deeply to cover each area. A trace pipeline carries trajectories from the runtime layer into the evaluation store and applies sampling rules and field masking in transit. The evaluation store versions datasets, rubrics, judge versions, results, and autonomy grant records, so every score stays traceable to the criteria that produced it. Versioned test assets are an inherited discipline, and the store keeps its conventions.

Three entries are new to that discipline: rubrics, judge versions, and autonomy grant records.

Rubrics stand in for expected outputs because agents allow many valid answers. Judge versions pin a grader whose behavior shifts across upgrades. Autonomy grant records appear because scores support standing permissions, not one-time releases. Governance policy sets retention, and platform engineering operationalizes it. Records supporting past autonomy grants stay under audit retention, and records under regulatory hold remain outside the archive.

The scoring plane produces the scores. Scoring rubrics state how to judge each case, so scores stay reproducible and comparable as datasets and policies evolve. Judge services expose rule-based checks, model-based judges, and human review queues behind one scoring interface. Judge services read production trajectories, so they inherit the data handling constraints of the systems that produced them, including residency and access controls.

The decision plane turns scores into verdicts. A regression harness runs suites against candidate configurations and computes statistical comparisons against baselines. Continuous monitoring scores production behavior over time and detects quality drift before users report it.

The governance plane records what the enterprise decided and why. Autonomy grant records bind each permission decision to the evidence behind it. Governance reporting publishes trends, gate outcomes, and autonomy decisions to engineering and governance audiences.

Privacy controls and cost governance are cross-cutting concerns rather than capabilities of their own. Privacy controls apply at trace capture, inside the store, and within judge services. At capture, the pipeline redacts or tokenizes sensitive fields against a policy governance approves, so production data reaches the store and judges only in masked form. Cost governance applies to sampling rates, judge tokens, and human review time, with budgets per agent and alerts on anomalous evaluation spend. Naming them as cross-cutting keeps every team responsible for them, rather than one team alone.

Figure 1 paired each problem with the capabilities that answer it.

The arrangement mirrors the separation of concerns running through this series, where memory determines what agents retain, context engineering shapes what they see, and runtime governs what they do. Evaluation shows how the enterprise knows, and keeps knowing, that those three layers behave as intended. Sections 5 through 9 open the four planes in turn, with the decision plane spanning Sections 7 and 8. Section 10 assigns ownership of the layer, and Section 11 routes its findings back into the rest of the stack. A short orientation on what evaluation points at comes first.

4. Three evaluation surfaces

The planes describe the machinery. What that machinery points at needs its own answer because evaluation programs often stall when teams disagree about what to evaluate. Separating three surfaces resolves much of the confusion and tells each capability in Figure 2 where to point.

The model surface measures the underlying model in isolation. Capability, robustness, and safety benchmarks live here. Vendors and research groups publish most of this evidence. Enterprises should consume it, replicate the parts that matter, and treat it as necessary background, not as sufficient assurance.

The agent surface measures the agent as configured, meaning the model plus its prompts, tools, memory, and guardrails. Relevant measures include task completion rates, tool call correctness, policy adherence, and the quality of full trajectories. This surface holds the configuration the enterprise controls, and it deserves most of the evaluation investment.

The system surface measures end-to-end outcomes across the full stack. Retrieval quality, memory behavior, orchestration across multiple agents, latency, and cost all shape what users experience. A capable agent inside a weak retrieval pipeline still fails the business. The scenario in Section 1 sat mostly on the agent and system surfaces, while published evidence concentrates on the model surface.

Practitioners also describe evaluation methods as black box or white box, and the two framings fit this map [2]. Black-box evaluation observes external behavior against statistical thresholds, which is how the scoring plane measures the agent and system surfaces. White-box evaluation examines internal reasoning, planning, and tool routing, which trajectory-level analysis performs on the same surfaces. System-level assurance leans on black-box methods, and component-level assurance leans on white-box methods. The surfaces state what evaluation points at, and the method framing describes how deeply it looks.

The surfaces also organize the metrics. A balanced program measures technical behavior and business results together because an agent can score well on one and fail the other. Six categories group the common measures, and Figure 3 maps each to the surfaces it describes.

Figure 3. Six metric categories

Source: Infosys

The business outcomes category carries particular weight. Technical scores tell an enterprise how the agent behaves. Cost per completed task, escalation rate, and customer satisfaction tell it whether the agent earns its place. Tracking these measures shows early whether an agent supports its business case.

5. The data plane: Evaluation datasets as the core component

This plane comes first because every other capability inherits the quality of the evaluation dataset. A calibrated judge scoring unrepresentative cases returns precise numbers about the wrong thing. A statistical gate on a thin suite reports confidence the evidence does not support. The governance record built on either one then documents a decision the enterprise may not be able to defend. Most evaluation problems that enterprises describe trace back to a dataset problem.

The work also resists shortcuts. Datasets need domain expertise to build, ownership to maintain, and discipline to keep clean. This section covers what a strong dataset looks like, where cases come from, how to implement the store that holds them, and who governs the result. A fixed property list makes the term strong testable, and the first subsection opens with it.

5.1 Characteristics of a strong evaluation dataset

Eight properties separate a dataset that supports autonomy decisions from a collection of test prompts. Each property carries a test to apply, and each has a weaker pattern that commonly appears in its place. Representative coverage, risk weighting, provenance, and versioning extend test design disciplines that predate agents. Grading, trajectory awareness, adversarial coverage, and contamination defense change in kind, for reasons Section 2 named.

Representative coverage matches the case mix against recent production traffic by intent and channel, rather than drawing cases from the product specification alone. Risk weighting gives high-consequence intents deeper coverage and stricter pass bars, instead of uniform coverage across intents of unequal consequence.

Provenance records source, author, approval date, and review status on every case, which prevents cases of unknown origin that nobody is willing to retire. Versioning permits change only by new version, and every score cites the version it ran against, which rules out silent edits that make historical scores incomparable.

Grading lists acceptable outcomes and disqualifying content for each case, in place of a single expected output that fails valid paraphrases. Trajectory awareness states the required tool calls, approval points, and forbidden actions, where weaker suites record only the final answer. Adversarial coverage includes instruction override attempts, ambiguous authority, and out-of-scope requests, where weaker suites test only cooperative users. Contamination defense runs scheduled scans that match cases against prompts, training corpora, and retrieval indexes [1]. The scans catch evaluation cases pasted into a prompt library as few-shot examples.

A worked case record shows those properties in one place. The example returns to the claims appeal scenario from Section 1. The same shape applies to lending decisions, service provisioning, or any workflow where an agent states policy and takes action. Figure 4 lists the record in full.

Figure 4. Claims appeal evaluation case record

Source: Infosys

The rubric identifier follows a namespace convention—artifact type, domain, rubric type, and version—so that each score traces to the exact criteria that produced it. Two fields do most of the work. The required trajectory field turns the case into a test of the path rather than the answer, which addresses failures inside the trajectory. The disqualifying content field gives the judge a bright line, which produces more consistent scores than a general quality instruction and addresses the absence of a single reference answer.

A case class extends this record with variations on entities, phrasing, and channel. One reviewed incident protects against a pattern rather than a single input. Domain owners approve each case class because different groupings catch different failure patterns and carry different false alarm rates.

5.2 Where cases come from

Cases come from three sources. Golden datasets come from real workflows. Domain experts curate representative tasks with graded reference outcomes, including acceptable variations. These sets are expensive to build and remain the anchor of any program. A few hundred well-chosen cases usually carry more information than thousands of shallow ones.

Synthetic datasets extend coverage into rare territory. Generators produce edge cases, adversarial phrasings, and unusual entity combinations that production has not yet surfaced. Seed the generator from the intent taxonomy and the risk classes, so that coverage follows the consequence of failure, not convenience. Every generated case needs expert approval before gating a release. Teams should also track the share of synthetic cases in each suite because a suite dominated by generated text drifts from operational language.

Production traces close the loop. Runtime systems already record agent trajectories for observability. Failed or escalated trajectories, once reviewed and labeled, become regression cases. Successful trajectories on novel tasks become new coverage.

A worked example shows the pipeline. The insurer from Section 1 updates its appeals policy, and the agent quotes the superseded appeal window to a customer. Triage pulls the trace, and a reviewer labels the cause as stale retrieval, not faulty reasoning. The case enters the suite with fixtures pinning the policy version and the effective date. The case class then varies the policy clause, the effective date, and the phrasing of the question. The suite now covers the failure pattern rather than the single incident.

5.3 A recommended technical implementation

Enterprises ask what to build. The following implementation uses tools most engineering organizations already run, and teams adapt it to the stack they have.

Store cases as versioned, human-readable records in the same version control system that holds application code, with a schema validated in continuous integration. Case review then follows the pull request path teams already trust, and the same workflow records the approval history. Keep large fixtures and trace payloads in object storage, referenced from the case record by content hash. The case stays reviewable in a diff, and the payload stays out of it.

Register each suite in a manifest that names the owner, the risk class, the rubric version, and the run policy. Tier the suites by cost and purpose. A smoke tier of 30 to 50 cases runs on every commit and returns within minutes. A gate tier of several hundred cases runs before release. A full tier, including rare and adversarial classes, runs nightly and before any model upgrade. Tiering keeps fast feedback affordable and reserves expensive coverage for decisions that warrant it.

Run contamination scans on a schedule against prompt libraries, training corpora, and retrieval indexes. Each scan needs a named owner and a documented action when a match appears. That action is usually retirement of the affected case and replacement from a fresh source. Emit every result as a record that cites the case version, rubric version, judge version, configuration hash, and run timestamp. Scores missing those fields may not be able to support an autonomy decision months later.

Give the dataset a retirement policy as well. Cases that no longer reflect a live workflow leave the suite through a deliberate decision. Figure 5 shows the resulting life cycle, including the path that carries production incidents back into the dataset.

Figure 5. The evaluation dataset life cycle

Source: Infosys

5.4 Governing the dataset

Someone must own each dataset and approve changes to it. Teams must track dataset versions because a score means nothing without the version that produced it. What counts as the same failure is also a governed definition, which domain owners approve so that it stays consistent across tickets.

Dataset health deserves its own reporting. Useful measures include coverage against production intents, the share of synthetic cases, and the age distribution of cases. Contamination scan results belong there too, alongside the count of cases that pass and fail inconsistently across repeated runs. A monthly review of these measures surfaces decay while correction remains inexpensive. Enterprises apply this discipline to financial data as a matter of course. Evaluation data warrants the same treatment because autonomy decisions rest on it.

6. The scoring plane: Rubrics, judges, and human review

A dataset only states what to test. This plane sets what counts as passing. Three mechanisms carry different strengths, and judge services expose all three behind one interface.

Rule-based checks verify objective properties. Did the agent call the required tool? Does the output parse, and does it omit restricted data? These checks are cheap, fast, and unambiguous. They should run on every trajectory, in testing and in production. They confirm the shape of a response and leave its meaning unjudged, so a trajectory can pass every rule and still answer incorrectly.

Human review verifies judgment-heavy properties. Tone, appropriateness, and domain nuance still exceed automated scoring in many settings. Human review is slow and costly, so programs should reserve it for calibration, high-risk actions, and disputed cases.

Model-based judging sits between the two. A second model reads the agent's output and scores it against a written rubric. One rubric line asks whether the response states the correct appeal window for this policy type.

The judge answers each line and returns a score. This method scales far beyond human review and handles nuance that fixed rules miss.

An agent-based variant extends model-based judging to full trajectories. A judge agent replays the case with ground-truth context and assesses the plan, each tool call, and the recovery path [2]. The variant suits trajectory-heavy workflows, and it inherits every safeguard this section applies to model-based judges.

The rubric matters as much as the judge. Vague criteria produce noisy scores from any judge, and precise criteria produce consistent ones. Four criteria drawn from the claims scenario show the difference, and Figure 6 pairs each vague criterion with its precise replacement.

Figure 6. Vague scoring criteria and their solution

Source: Infosys

Teams should draft rubrics with domain owners, apply the care they give application programming interface (API) contracts, and version them alongside the datasets they score.

Model-based judging also needs safeguards because the judge is itself a model. Judges carry position and verbosity biases, prefer certain writing styles, and change behavior when their underlying model receives an upgrade [3]. Calibration provides the safeguard. Teams score a sample of cases with both the judge and human experts, then compare the two sets.

Calibration works best as a defined procedure. Draw a stratified sample that covers every risk class, then score it with the judge and with a human panel. Report agreement per class using a statistic such as Cohen’s kappa [4]. Set the required agreement per risk class, record it as policy, and name the owner who approves changes to it. Wide disagreement means the judge needs repair before its scores gate a deployment. Calibration must run on a schedule, with documented results.

The safeguard inherits the quality of the human panel, so the panel needs domain expertise and its own consistency checks. It also has a stated limit. Agreement on the calibration sample does not extend to cases outside its distribution, so the sample must include the case classes that gate autonomy. Every score records the judge version that produced it. A score from an upgraded judge is not comparable to its predecessor until calibration confirms the two agree.

7. The decision plane: Regression for nondeterministic systems

The decision plane turns scores into verdicts, and the regression harness is its first instrument. Every change to an agentic system raises the same question: did anything break? Prompt edits, tool updates, and model upgrades all carry regression risk. Deterministic diffing cannot answer the question, so the regression harness has to become statistical. Figure 7 shows the resulting flow.

Figure 7. Statistical regression flow

Source: Infosys

Repetition replaces single runs. Each case runs several times, and the pass rate becomes the measurement. A case that passes 19 of 20 runs differs meaningfully from one that passes 12 of 20, even though both fail sometimes.

Pass thresholds take the place of binary verdicts. Teams define acceptable pass rates per case class, with stricter bars for high-risk actions. A release gates on aggregate movement against the previous baseline, within a stated confidence interval [5]. That release gate checks each high-risk class against its own bar before it weighs the aggregate.

A worked example shows why the order matters. A suite of 200 cases runs 20 times each, producing 4,000 runs. The baseline aggregate pass rate is 94.5%, and the candidate scores 93.8%. That difference sits inside the confidence interval the team stated, so the aggregate check passes. The high-risk appeals class tells a different story. Its pass rate moves from 19 of 20 to 15 of 20, which falls below the 90% bar the owner set for that class. The per-class check blocks a release that the aggregate would have approved.

Thresholds carry the same life cycle discipline as rubrics. Each one has a named owner who approves changes and tests boundary conditions. A 95% bar remains ambiguous until the team states whether a measured 95.0% passes and which confidence interval applies. The harness enforces thresholds, and a separate governance loop owns their correctness.

Comparison also happens at the trajectory level. Two runs can produce different valid outputs through different valid paths. Comparing tool call sequences, step counts, and escalation behavior often reveals regressions that final outputs hide. Consider a model upgrade after which the agent takes twice as many steps to reach the same answers. The outputs still pass, and the system now costs more, runs slower, and follows unfamiliar paths. Trajectory comparison catches this change, while output comparison misses it.

Vendor upgrades have always warranted a regression pass, and model upgrades follow the same playbook: freeze the datasets, run the full suite on both versions, and compare the distributions. Review divergent trajectories by hand before promotion.

8. The decision plane: Certification and continuous evaluation

Continuous monitoring completes the decision plane. Two operating models dominate current practice. The gate model certifies an agent before deployment. The monitor model evaluates live behavior continuously. Enterprises need both because each covers the other's blind spot.

Preproduction certification answers a bounded question: does this agent, in this configuration, meet the bar today? Gates work well for major releases, new tools, and expanded permissions. Their weakness is decay. Models receive upgrades, tools change their behavior, data distributions shift, and users find new phrasings. Continuous operation compounds the problem because agents make judgment calls around the clock that no release gate reviewed. The insurer in Section 1 met this failure directly, when two silent model upgrades separated the running agent from the certified one.

Continuous evaluation treats production as the permanent test bed. The runtime layer described earlier in this series emits trajectories as a byproduct of normal operation. Sampling those trajectories, scoring them with rules and judges, and trending the results turns observability data into ongoing verification. Teams set alert thresholds on these scores. A score that falls below its threshold notifies the owning team, the same way a rising error rate notifies the owners of a conventional service.

Score decay has two possible sources: the agent changed, or the judge changed. Before an alert triggers a governance action, the team reruns the calibration set on the current judge. Stable calibration results direct the investigation at the agent. Shifted results mark the judge as the source, and the agent keeps its standing.

Sampling policy is where cost governance becomes concrete. A service handling 50,000 trajectories a day may not be able to send all of them to a model judge. A workable policy scores every trajectory with rule-based checks, then routes a small percentage to a model judge. Every high-risk action and every escalation goes to human review. Teams then set the sampling rate as an explicit cost versus coverage decision, with budgets per agent and alerts on anomalous evaluation spend.

False alarms belong in the same ledger. Alert tuning is an inherited operations discipline. The stakes are higher because human review now serves as a primary safety mechanism for systems that raise alerts at machine speed. An alert threshold that fires on noise consumes investigation time. Teams meter alerts that resolve without a defect, and tune thresholds against the review cost they generate.

The two models also exchange material. Production failures become new certification cases. Certification suites define the scoring logic that continuous evaluation reuses. Treating them as one architecture with two entry points avoids duplicated tooling and inconsistent standards.

9. The governance plane: Evidence bars and autonomy levels

The governance plane closes the architecture. Evaluation earns its governance role when scores map to permissions. Established risk frameworks separate measurement from governance for the same reason, and treat oversight as a function spanning the life cycle instead of a release checkpoint [6]. A three-level autonomy model makes the mapping concrete, and each level carries a rising evidence bar. Figure 8 summarizes the three levels and what each one requires.

Figure 8. Rising evidence bars across three levels of agent autonomy

Source: Infosys

At the advisory level, the agent drafts, summarizes, and recommends while humans decide and act. Errors consume review time and rarely reach customers, so the bar admits some variance. Under supervised execution, the agent acts and a human approves the action or samples completed work. Approval queues also generate labeled data, which feeds the datasets described in Section 5.

Autonomous execution lets the agent act without per-action review. This grant requires the full architecture: continuous evaluation with alert thresholds, documented judge calibration, tested rollback runbooks, and scheduled recertification. High-risk action classes remain at the supervised level even when the agent holds autonomous status elsewhere.

Each grant becomes a recorded artifact. The record binds the decision to the dataset version, rubric version, score set, calibration report, and approving authority at the moment of approval. When an auditor later asks why an agent acted alone, the enterprise reconstructs the full decision, with the scores in their original context. The evaluation store holds this record alongside the evidence that produced it. Grant records carry audit weight, so they need the same integrity protections the runtime applies to execution records. This artifact is the architecture's answer to evidence detached from authority.

Autonomy grants attach to a single agent, and workflows increasingly compose several. A supervised agent that invokes an autonomous one raises the question of which bar governs the composed action. The conservative rule holds: a composed workflow runs at the most restrictive level among its participants. A team that wants the composition to run at a higher level certifies the composed system itself, with its own grant record and evidence.

Movement between levels works in both directions. Promotion follows sustained evidence at the higher bar. Demotion follows score decay below the current bar, confirmed against a stable judge, with permissions returning to the previous level pending review.

Demotion needs a defined enforcement path. The evaluation layer decides the autonomy level, and the runtime enforces it. A demotion signal travels to the runtime as a policy change. The runtime updates the agent's permissions, records the change in its audit trail, and applies it within a stated time bound. Workflows already in-flight complete under supervised review rather than under the revoked grant. A high-severity demotion can instead halt in-flight work, and the policy states which severities halt and which complete under supervision.

The trigger mode is part of the contract. Automatic demotion acts within minutes and risks flapping on a noisy score. Alert-driven demotion inserts a human decision and accepts the added latency. Teams choose the mode per action class and record the choice as policy.

The life cycle also needs defined ends. An onboarding criterion states the minimum evidence an agent presents before its first grant, which keeps unsanctioned agents out of production [2]. An offboarding criterion states when the enterprise retires an agent, such as sustained decay, a withdrawn business case, or an unsupported model version. Retirement revokes permissions, archives the grant history under audit retention, and removes the agent from the production inventory.

10. Ownership and operating model

Sections 5 through 9 detailed the machinery. Architecture without clear ownership falls into disuse. Three organizational questions decide whether evaluation persists.

The first is placement. Some enterprises place agent evaluation within quality engineering. Others build the capability inside the AI platform team. Both work. Assigning evaluation to each product team independently works poorly because it produces inconsistent standards and duplicated harnesses. A central evaluation function with embedded domain reviewers balances consistency with context.

The second is authority. Evaluation results must connect to decisions. The function needs a documented mandate: which scores gate a release, who signs off on movement between autonomy levels, and what evidence supports each grant. The graded model in Section 9 gives that mandate its structure.

The third is escalation. When continuous evaluation detects decay, someone must own the response. Runbooks should define who investigates, what rollback options exist, and when to restrict an agent's permissions pending review. The demotion path in Section 9 supplies the permission side of that response. These duties resemble incident management for conventional services, and enterprises adapt structures they already trust.

11. Closing the improvement loop

Evaluation findings serve a second purpose beyond the gate. Each finding points at a specific layer of the stack this series has described. Routing the finding to that layer turns evaluation into a driver of improvement as well as a control. The routing follows the failure signature, and six signatures cover the common cases. Figure 9 routes each signature to the layer that owns the fix.

Figure 9. Failure signatures

Source: Infosys

The last signature deserves attention because it points back at the core component. A suite that passes while users complain has a coverage problem, and no amount of judge tuning corrects it.

The mechanics already exist. The trace pipeline carries the evidence, and the evaluation store preserves the failing case. The addition is a triage practice that assigns each finding an owner in the corresponding layer. The same suite that caught the failure then verifies the fix, which closes the loop and hardens the system against recurrence.

12. Conclusion

Enterprises earned decades of trust in software through disciplined verification. Agents deserve the same discipline in a form suited to their nature. No engineering failure drove the scenario in Section 1. The organization lacked current evidence about behavior it had already put in front of customers.

The shift is architectural: from testing as a phase to evaluation as a permanent layer, with its own data, scoring services, regression methods, and owners. Evaluation datasets sit at the center of that layer, and programs that underinvest there find every downstream measure unreliable. Enterprises that build this layer expand agent autonomy as evidence supports it. Where the layer is missing, agent behavior surfaces through incidents instead. The work is substantial, and it draws on skills quality teams already hold. Extending those skills to probabilistic systems is the next stage of enterprise quality engineering, and the autonomy decisions ahead will rest on the evidence it produces. Six recurring problems define the need, seven capabilities across four planes answer it, graded autonomy converts scores into governed permissions, and the improvement loop routes each finding back to the layer that owns the fix. The result is a system that earns trust continuously, not once.

References