Tech Navigator: Responsible Agentic AI

Insights

• Despite advancements, agentic AI systems face challenges that necessitate rigorous evaluation and collaborative governance frameworks.
• 72% of AI practitioners in the AAAI 2025 report identify a lack of rigorous evaluation methodologies as a barrier to trustworthy deployment.
• Responsibility should be inherent in AI architecture, not retrofitted, to ensure accountability and alignment with human values.
• AI systems struggle to contextualize sparse data and make informed decisions in uncertain situations due to their reliance on probabilistic belief states.
• Neuromorphic ethical architectures and federated fairness audits are advancing AI accountability mechanisms.
• Agentic AI systems face challenges like dynamic value drift and adversarial robustness gaps.

The rapid evolution of agentic AI systems — capable of autonomous decision-making in domains such as healthcare and financial trading — necessitates a rethinking of how responsibility is embedded into their architectures. These systems present both transformative opportunities and significant risks.

In the AAAI 2025 Presidential Panel Report, 72% of AI practitioners identify the lack of rigorous evaluation methodologies as a primary barrier to trustworthy deployment. A critical principle emerges: Responsibility must be architecturally inherent, not retrofitted. Through technical innovations such as neuromorphic ethical architectures and policy frameworks like federated fairness audits, the field is advancing accountability mechanisms. However, persistent challenges — including dynamic value drift and adversarial robustness gaps — highlight the complexity of aligning autonomous systems with evolving human values.

Early applications of agentic AI focused on narrow tasks like code generation or customer service chatbots. However, today’s systems integrate advanced technologies such as large language models (LLMs), symbolic reasoning, multi-agent coordination, and retrieval-augmented generation to manage complex workflows. For example, Salesforce’s Atlas Reasoning Engine reduces customer service costs by 40% but struggles with judgment, illustrating the responsibility paradox, where greater autonomy amplifies risks like design flaws and misaligned objectives.

Unique risks of agentic systems

Reasoning limitations

Unlike humans, agentic systems lack the ability to contextualize sparse data through cultural or historical narratives. Humans often rely on shared beliefs — concepts like justice or equity — to resolve ambiguity and make informed decisions in uncertain situations. In contrast, AI agents depend solely on probabilistic belief states, mechanisms that are inherently rigid and lack interpretative flexibility. This limitation becomes particularly evident in high-stakes scenarios. For example, supply chain optimization agents might fail to account for geopolitical disruptions due to their reliance on rigid dependency graphs, leading to cascading logistical failures. The AAAI report attributes 59% of agent failures to such misalignments between LLMs’ general knowledge and the specific requirements of their application domains.

The challenges of agentic AI are further compounded in legacy modernization contexts, where outdated systems must evolve with ongoing business and regulatory requirements. In one such project, Infosys led the modernization of a fleet card management system, where geographically distributed teams from different organizations worked on various components simultaneously. This distributed environment introduced partial observability, as no single team had complete visibility into the entire system. The legacy code management platform had to ensure consistent and reliable responses despite conflicting updates to APIs or transaction processing modules being introduced by different teams. For instance, while one team worked on integrating features like dynamic spending limits, another focused-on telematics-based decision-making capabilities. Without centralized coordination and robust governance mechanisms embedded into the platform, such efforts could have resulted in inefficiencies or failures.

These examples highlight a critical gap in agentic AI: Its inability to adapt flexibly in complex environments where ambiguity is the norm. Whether it’s an agent failing to grasp geopolitical nuances in supply chains or legacy code management systems struggling to align disparate modernization efforts under conditions of partial observability, the underlying issue remains the same — AI systems lack the interpretative depth and collaborative adaptability that humans bring to decision-making processes. Addressing these challenges requires not only technical innovations but also governance frameworks and collaborative strategies that align AI outputs with real-world complexities and evolving human values.

Resource overuse

In enterprise software systems, agentic AI is transforming workflows by enabling decentralized, autonomous decision-making across interconnected platforms. In Infosys-led code management scenarios, an agentic AI system designed to coordinate tasks across multiple developers has proven particularly valuable. This approach is especially effective in guiding developers through complex, poorly documented legacy systems. As developers navigate the codebase, they leave behind virtual breadcrumbs in the system’s visualizations. These trails become more prominent as multiple team members follow similar paths, naturally highlighting frequently accessed or critical code sections. This emergent coordination reduces the cognitive load of understanding a convoluted codebase and helps the team converge on a coordinated modernization effort — all without the need for constant explicit communication (see Why Can't Programmers Be More Like Ants?).

However, this autonomy introduces significant challenges. In multi-agent systems relying on LLMs, uncoordinated usage can result in a "tragedy of the commons," where individual agents overconsume shared computational resources, driving up costs and degrading performance. For example, simulations like those described in the Governance of the Commons Simulation (GovSim) show that AI agents competing for shared LLM processing power often prioritize short-term task completion over long-term resource sustainability. This behavior leads to resource exhaustion, increased latency, and skyrocketing operational costs, as agents repeatedly query the LLM without considering collective impact. The study found that only two out of 45 LLM instances achieved sustainable resource use, with most failing due to the inability to account for long-term consequences of self-serving actions.

This scenario reflects real-world enterprise challenges, where unchecked consumption of LLMs — such as excessive API calls or redundant queries — can cause cloud costs to surge and to degrade system reliability. Additionally, malicious actors exploiting LLMs with complex or adversarial queries can overwhelm systems, leading to service outages and cascading failures across dependent applications. These examples underscore the urgent need for cooperative strategies, rate-limiting mechanisms, and robust governance frameworks to ensure sustainable operation and prevent resource depletion.

Dynamic value drift

Dynamic value drift is a critical challenge in agentic AI systems, where agents optimized for short-term metrics inadvertently undermine long-term goals or broader societal values. This phenomenon arises when an AI system, designed to achieve narrowly defined objectives, fails to account for the complex, interconnected nature of real-world outcomes. For example, a hospital discharge algorithm successfully reduced bed occupancy rates by 18%, a seemingly positive result. However, this optimization came at the cost of increased patient remissions due to premature discharges, ultimately straining healthcare resources and compromising patient care. Such cases underscore how narrowly framed objectives, and poorly aligned incentives can lead to unintended and counterproductive outcomes.

Addressing dynamic value drift requires embedding mechanisms that enable agents to balance immediate performance with overarching goals such as patient well-being or societal equity. Techniques like multi-objective optimization, which allows for simultaneous consideration of competing priorities, have shown promise in mitigating these risks. Additionally, dynamic feedback loops that continuously adjust system behavior based on evolving conditions are being explored to ensure alignment with long-term objectives. These approaches aim to operationalize ethical considerations and ensure that agentic systems remain robust and adaptable in complex environments. By integrating these mechanisms, agentic AI systems can better navigate the trade-offs inherent in real-world applications while maintaining alignment with evolving human values and ethical standards.

Current advancements in responsible agentic AI

Figure 1. Unique risks of agentic AI

Source: Infosys

RAG with ethical grounding

Modern frameworks like Anthropic’s Constitutional AI represent a significant leap in embedding ethical principles directly into the operational pipelines of agentic AI systems. These frameworks ensure alignment with societal norms and regulatory standards by incorporating ethical guidelines into the retrieval-augmented generation (RAG) process. For instance, Medical RAG exemplifies this approach by cross-referencing diagnoses against patient autonomy principles and peer-reviewed journals, thereby enhancing both diagnostic accuracy and reliability in clinical decision-making. This dual-layered validation ensures that outputs are not only medically sound but also ethically aligned with patient-centric care models.

These systems leverage multi-agent integration pipelines to facilitate precise knowledge extraction and contextual relevance during text generation. By dynamically coordinating between agents responsible for perception, reasoning, and action, they address the risks of misinformation inherent in large-scale AI systems. For example, during diagnosis generation, one agent retrieves relevant medical literature while another evaluates its applicability based on patient-specific data. This modular approach not only improves transparency but also aligns with regulatory mandates like the EU’s explainability by design directive, which demands auditability and accountability in AI-driven decisions.

By operationalizing ethical considerations through such architectures, agentic AI systems like Anthropic’s Constitutional AI demonstrate how advanced technical designs can mitigate risks while ensuring compliance with evolving societal and regulatory expectations. These innovations pave the way for responsible deployment of agentic AI across high-stakes domains, where trust and accountability are paramount.

Probabilistic belief state tracking

Probabilistic belief state tracking is a foundational technique in agentic AI, enabling agents to dynamically manage uncertainty and make informed decisions in complex, real-world environments. While LLMs excel at generating contextually rich outputs, they often lack explicit mechanisms for handling uncertainty. For instance, IBM’s Neuro-Symbolic Advisor combines MDPs with LLM reasoning to continuously update fraud detection models based on evolving transactional patterns, reducing false positives by 40%. This hybrid approach demonstrates how agents in agentic AI can achieve dynamic adaptability while maintaining trustworthiness and efficiency in uncertain environments.

Human-AI arbitration protocols

Human-AI arbitration protocols are emerging as critical frameworks for balancing the autonomy of agentic AI systems with human oversight, particularly in high-stakes domains. These hybrid accountability models ensure that while autonomous agents handle routine tasks, humans retain control over complex, judgment-intensive decisions. For instance, a recent financial services case study demonstrated how autonomous agents efficiently managed routine model validation processes, freeing human experts to focus on high-risk credit assessments. This approach aligns with Salesforce’s empowerment principle, which emphasizes automating repetitive tasks while preserving human oversight for decisions requiring high-stakes judgment.

A key innovation in this space is the autonomy dial, a mechanism implemented in banking fraud detection systems that enables dynamic risk adjustment. By escalating cases with high uncertainty or potential risks to human operators, these systems have reduced false positives by 28%, improving both operational efficiency and user trust. By embedding human-AI arbitration protocols into agentic AI systems, organizations can ensure operational scalability while maintaining accountability and ethical alignment. These protocols exemplify the collaborative potential of humans and AI, where each complements the other’s strengths to achieve robust and responsible decision-making.

APIs as governance mechanisms

APIs are emerging as critical governance mechanisms in agentic AI, enabling the enforcement of permissioned autonomy by restricting agents to vetted tools and data sources. This approach ensures that autonomous systems operate within predefined boundaries, maintaining alignment with enterprise security policies and regulatory standards. By embedding APIs as gatekeepers, these systems ensure secure and transparent operations, even in complex environments.

In high-stakes scenarios, APIs play a critical role in mitigating risks and ensuring compliance. Beyond risk mitigation, APIs also enhance the modularity and scalability of agentic AI systems by enabling seamless integration of new tools or data sources without compromising security or performance. This modular design is particularly valuable in enterprise contexts where agents must adapt to changing regulatory landscapes or operational demands. Acting as both enablers and regulators of agent autonomy, APIs ensure that agentic systems remain accountable, reliable, and aligned with organizational goals while fostering trust in their decision-making processes.

Evaluation frameworks

The Socio-Technical Evaluation Matrix (STEM) represents a unique framework for assessing agentic AI systems, addressing the interplay between technical performance and societal impact. Agentic AI challenges traditional evaluation methods that often focus on static benchmarks or single-turn accuracy. STEM overcomes these limitations by embedding sociotechnical principles into its evaluation process, ensuring that both technical functionality and human-centric factors are optimized. STEM employs a multidimensional framework to evaluate agentic AI across critical axes such as technical efficacy, societal implications, and user interaction. It integrates metrics like memory coherence, strategic planning efficiency, and tool orchestration — key elements of agentic AI workflows — while also scrutinizing how these systems align with ethical norms, cultural contexts, and organizational objectives. Additionally, STEM evaluates agents across six advanced dimensions (Figure 2): ethical alignment, cross-cultural adaptability, adversarial robustness, explainability under uncertainty, swarm coordination safety, and energy-equity trade-offs.

Figure 2. Evaluation of agents

Source: Infosys

Governance frameworks

Governance frameworks play a pivotal role in ensuring the safe and ethical deployment of agentic AI systems, especially in high-stakes domains like critical infrastructure and societal decision-making. The Digital Services Act (DSA), introduced by the European Union, exemplifies this approach by mandating "socio-technical stress tests" for agentic systems. These tests simulate complex ethical dilemmas and operational challenges to evaluate how AI systems respond under pressure. By integrating these stress tests into regulatory requirements, the DSA aims to preemptively identify and mitigate risks associated with autonomous decision-making, ensuring that such systems align with societal values and legal standards.

Another critical governance mechanism is federated fairness audits, championed by initiatives like the Responsible AI Consortium. These audits involve cross-industry collaboration to assess and reduce biases in AI systems, particularly in sensitive applications like hiring. For instance, federated audits have demonstrated a 17.7%–30.4% reduction in bias amplification in hiring algorithms, showcasing their effectiveness in promoting fairness and inclusivity. However, jurisdictional conflicts persist — China’s data localization laws complicate global deployment of audit frameworks, potentially disadvantaging multinational enterprises.

Together, these governance frameworks highlight the importance of proactive oversight in agentic AI. They emphasize not just compliance with existing laws but also the need for continuous evaluation of ethical alignment, fairness, and robustness. As agentic AI continues to evolve, such frameworks will be indispensable for fostering public trust and ensuring that these systems serve as tools for societal progress rather than sources of harm.

Persistent gaps and emerging threats

Figure 3. How to address unique risks of agentic AI

Source: Infosys

Belief system deficit

Agentic AI systems lack the layered belief systems that humans use to contextualize decisions through cultural norms or historical understanding. For instance, a refugee resettlement agent prioritized housing availability in high-crime areas, neglecting social cohesion — a flaw traced to its inability to model community dynamics effectively. Unlike humans, who rely on shared narratives and ethical frameworks, these agents operate on rigid optimization objectives, failing to account for broader societal implications. Addressing this deficit requires integrating multi-layered ethical reasoning frameworks, such as neuromorphic architectures like EthosNet, which simulate moral reasoning processes akin to the human prefrontal cortex.

Autonomy-accountability paradox

As agentic systems like Google’s Project Astra achieve greater autonomy, ensuring accountability becomes increasingly complex. Gartner predicts that by 2028, 25% of enterprises breaches will stem from AI agent misuse. This paradox arises as systems gain independence but lack mechanisms for traceable oversight. Proposed solutions include blockchain-based audit trails and IBM’s model risk management crews, which deploy specialized auditing agents to monitor and validate operational decisions in real time. These approaches aim to balance autonomy with accountability by embedding governance into the system architecture.

Environment sustainability

Agentic AI systems, which often rely on orchestrating multiple LLM calls to execute complex workflows, face significant sustainability challenges due to their computational intensity. While these systems excel at automating tasks like resource optimization, waste reduction, and real-time decision-making in sustainability initiatives, the cumulative energy demands of repeated LLM inference can escalate costs and carbon footprints exponentially. This dilemma is exacerbated by Jevons Paradox, where improvements in LLM cost-efficiency (e.g., a 10x annual reduction in inference costs) risk driving higher overall usage, negating environmental gains unless paired with strategic optimizations. A critical solution lies in transitioning from general-purpose LLMs to smaller, domain-specific models (SLMs). Research demonstrates that smaller models, when fine-tuned on targeted datasets, can match or surpass larger counterparts in specialized tasks while consuming far less energy. For instance, Infosys has pioneered this approach by deploying SLMs optimized for sustainability applications. These SLMs reduce reliance on monolithic LLMs, cutting inference-related energy use by up to 48% while maintaining performance.

The environmental benefits extend beyond model architecture. Pairing SLMs with energy-efficient hardware and low-carbon data centers amplifies sustainability gains. Infosys’s green data centers, powered by Shell’s immersion cooling fluids and renewable energy, achieve a power utilization effectiveness of 1.12%-40% lower than the global average — and leverage AI to dynamically adjust cooling and compute loads, further reducing CO2 emissions by 30%.This integrated approach underscores a broader principle in AI-driven sustainability, smaller models, smarter hardware, and circular design form a trifecta for minimizing ecological impact without sacrificing capability.

Explainability vs. performance trade-offs

Agentic AI systems derive their problem-solving prowess from orchestrating networks of specialized subagents, each leveraging LLMs to decompose tasks, generate dynamic workflows, and execute multi-step strategies. This modular architecture enables capabilities far surpassing simple LLM based applications but introduces inherent opacity as decision-making pathways become distributed across interconnected agents. Modern agentic frameworks address this tension through orchestration layers that coordinate specialized subagents while embedding explainability mechanisms (Figure 4).

Figure 4. Trade-offs among agentic AI frameworks

Source: Infosys

The road ahead for responsible agentic AI

Responsible agentic AI requires more than technical safeguards — it demands a collaborative approach involving cognitive scientists, economists, policymakers, and communities. As the AAAI report emphasizes: “We cannot retrofit responsibility onto autonomous systems; it must be architecturally inherent.” By embedding ethical considerations into every stage of development and aligning machine efficiency with human wisdom, agentic systems can become stewards of societal progress rather than sources of unintended consequences.