As enterprises accelerate the deployment of autonomous artificial intelligence, a new and potentially serious governance gap is emerging. While organizations are investing heavily in AI agents to automate workflows and decision-making, many are losing visibility into accountability, especially in complex multi-agent environments.
New research from Korea. he points out the extent of the matter. In a survey of more than 400 IT and business leaders from large enterprises, 70 percent of respondents said they could detect when something had gone wrong, but could not identify which AI agent was responsible. This discovery highlights a key challenge in the evolution of enterprise AI. Here discovery is improving, but attribution is not.
Increasing complexity of many agents
Moving right multi-agent AI systems represents a natural progression of enterprise automation. Rather than relying on single models or worlds, organizations are deploying networks of specialized agents, each responsible for discrete tasks such as interacting with customers, processing data, or orchestrating decisions.
However, as these systems expand, their interactions become more and more obscure. When one agent triggers an action that passes through others, tracing the origin of an issue becomes difficult.
This is reflected in the survey findings. While most organizations have detection capabilities, the inability to isolate the source of a failure suggests that enterprises are operating AI ecosystems that they cannot fully explain.
The research also highlights a significant trust deficit. More than half (53 percent) of organizations admit they are running AI agents they don’t fully understand. This creates a paradox. Enterprises are relying on AI to deliver efficiency and innovation, but at the same time, they lack confidence in how these systems behave in real-world conditions.
In highly regulated sectors this raises immediate concerns. If organizations cannot explain how decisions are made, or why failures occur, regulatory compliance and risk management become significantly more complex.
Manual intervention remains the safety net
Despite the promise of autonomy, human oversight remains critical. The survey shows that 79 percent of enterprises have had to manually change autonomous AI actions, indicating that full automation remains some way off. Most importantly, these changes are not trivial. About 93 percent of respondents reported that corrective actions were costly and disruptive.
This suggests that organizations are effectively absorbing a form of operational drag, where the benefits of automation are offset by the cost of intervening when things go wrong.
From a systems perspective, this is not unusual. Complex systems tend to fail in unpredictable ways. What is changing, however, is the speed and extent to which these failures can propagate when driven by autonomous AI.
Discovery delays and customer-driven discovery
The findings also reveal variability in how quickly organizations can identify issues. About half of respondents said that failures are detected within one to four hours, while a third require four to eight hours. Detection methods are also uneven. For example, 39 percent say they rely on real-time dashboards, while 29 percent use automated alerts and 17 percent depend on log analysis.
Perhaps most troubling, 15 percent of enterprises rely on end users to report problems, effectively handing over discovery to customers. This poses a reputational risk. When customers identify failures before the organization does, trust can erode rapidly, especially in customer-facing services such as banking, retail or digital platforms.
Raj Koneru, CEO and founder of Kore.ai, describes this phenomenon as a form of “government debt”. The concept reflects how delays in identifying and resolving issues allow risks and costs to accumulate over time. In traditional IT systems, governance structures evolved alongside the technology. For AI, especially autonomous AI, deployment is often crossing the governance framework.
Visibility is not a responsibility
A key insight from the research is that observation alone is insufficient. Many organizations have invested in monitoring tools that provide visibility into system activity. However, these tools do not necessarily answer the critical question: who made the decision?
As Kore.ai Chief Marketing Officer Peter Mullen points out, there’s a difference between visibility and accountability. An AI agent that can be observed but not governed remains a risk. This distinction is particularly important as enterprises move towards AI-driven decision-making and introduce automated workflows with minimal human supervision. Without clear accountability mechanisms, these systems can become black boxes, visible but not controllable.
The findings point to a broader shift in how organizations should approach AI. Governance can no longer be treated as an afterthought. Instead, it should be involved in the design and deployment of systems from the start.
As organizations scale their use of AI, they must also face the challenges of control, trust and accountability. Perhaps deploying AI agents is no longer the hard part; it may be that their governance is.
