For the fifth year in a row, MIT Sloan Management Review and Boston Consulting Group (BCG) have assembled an international panel of AI experts that includes academics and practitioners to help us understand how responsible artificial intelligence is being implemented across organizations worldwide. In previous posts this year, we have explored artificial intelligence’s impact on the workforce, including why responsible AI requires more than training human experts to verify AI outputs.
This time, we asked our panel to react to the following provocation: Responsible governance that treats agents as autonomous decision makers will fail. On the surface, there is broad consensus, with a clear majority (72%) of our panelists agreeing or strongly agreeing with the statement. But digging deeper reveals a more nuanced conversation about what autonomy means, the relationship between autonomy and accountability, and what’s at risk when characterizing agents as “autonomous.” The picture that emerges: Calling agents autonomous decision makers risks allowing the humans and institutions behind them to evade responsibility for their actions. Effective governance, by contrast, ties every consequential decision back to a responsible party that can be held legally and morally accountable, in the context of a broader sociotechnical system.
Below, we share panelist insights and offer our practical recommendations for organizations thinking about agent autonomy through the lens of responsible AI governance.
Agents are increasingly “autonomous” — but only in an operational sense. Our experts acknowledge that agents are exhibiting a growing degree of technical or operational independence and ability to take action on their own. EnBW chief data officer Rainer Hoffmann says, “Agentic autonomy is real and growing,” while Renato Leite Monteiro, vice president of privacy, data protection, AI, and intellectual property at e&, observes that “self-improving agents are moving faster than we can map their failure modes.” Ben Dias, chief AI scientist at IAG, agrees that “agentic AI is rapidly moving beyond providing support or answers to taking autonomous action on our behalf.” For example, National University of Singapore vice provost Simon Chesterman points out that “agentic AI can plan, call tools, transact, and operate across workflows.” For these reasons, AI speaker and consultant Linda Leopold believes that “agents are autonomous decision makers, technically,” because “they act without constant human oversight and approval.”
But this kind of autonomy deserves closer scrutiny. Bruno Bioni, founder and director of Data Privacy Brasil, contends that “what looks like autonomy is [actually] delegated execution: selecting steps, using tools, acting within limits set by someone else.” Dias explains, “AI agents are given a goal and a set of guardrails, and then they independently determine and execute the sequence of actions required to achieve their given goal.” Amit Shah, CEO of Instalily.ai, similarly describes agents as “infrastructure that decides in the operational sense: It routes the order, moves the inventory, prices the risk, and so on.” As a result, Öykü Işik believes that “how we define autonomy in this context is critical.”
Operational autonomy does not translate into moral or legal accountability. For many of our experts, technical autonomy does not confer moral agency or responsibility. As Chesterman contends, “Autonomy in the engineering sense is not autonomy in the moral or legal sense.” Leopold cautions, “It gets problematic if we also start thinking of agents as autonomous in a moral sense — as entities with agency, or even coworkers, rather than the software systems they are.” Shah explains, “A machine can make the call, but it cannot own the outcome or consequence in the moral sense.” Jai Ganesh, Wipro’s former vice president of technology, agrees that “agents can make choices or execute actions, but they cannot be held accountable for the consequences.” For Carolina Aguerre, professor at Universidad Católica del Uruguay, “Responsibility is an inherently human faculty.”
Similarly, operational autonomy does not translate to legal responsibility. As Stanford CodeEx fellow Riyanka Roy Choudhury puts it, treating agents as autonomous decision makers “severs liability from capacity” since an agent “holds no assets to attach, no license to suspend, no deterrable interests.” Chow also points out that “agents have no legal standing and no assets,” adding, “They cannot be sued, pay damages, or be fully sanctioned.” Going further, Işik observes that “AI agents are stochastic and context-dependent,” not “coherent agents with stable intent” that meaningful accountability requires. For a recent example, Choudhury and others point to Moffatt v. Air Canada, in which a British Columbia tribunal rejected Air Canada’s argument that its chatbot was a separate legal entity accountable for its own misstatements. This case illustrates the challenge that companies face in governing agents that can act like human employees but cannot themselves be held morally or legally accountable.
Treating agents as autonomous decision makers undermines accountability. In fact, treating agents as autonomous creates an accountability vacuum and, as Bioni puts it, “imports a legal and moral status the technology has not earned.” As Chesterman cautions, “The more we speak as if agents ‘decide,’ the easier it becomes for firms and governments to launder responsibility through the machine: The model recommended, the agent acted, the human shrugged.” Or, as Bioni says, “it lets developers, deployers, and users hide behind ‘the AI decided’ whenever outcomes go wrong.” Even more bluntly, Shah calls the term autonomous decision maker “a governance fiction” that enables “blame laundering with better vocabulary.” For companies that remain accountable for the actions agents take, this accountability vacuum creates real risk if employees believe they can transfer blame and avoid responsibility.
The severity of this accountability vacuum depends on what’s on the line. For Monteiro, “Autonomy and accountability should not mix when the stakes are real” because regulators, boards of directors, and courts will require “a human they can hold responsible.” But while RAIght.ai co-CEO Richard Benjamins thinks that “impactful decisions should not be fully autonomously taken by AI agents,” he believes “trivial decisions can be.” Apollo Global Management’s AI lead Katia Walsh agrees that “for high-stakes decisions, responsible AI governance should not treat agents as autonomous decision makers, but for other contexts, it may be just fine” to treat them as if they were. And Aguerre contends, “Since not all AI agents perform activities with the same level of risk, the different levels of autonomy granted to an AI agent should be assessed against the tasks and object
