Companies using AI agents should develop and implement ethical standards for transparency, user education, and accountability and embrace responsibility for its impact.
In two recent lawsuits against the company Character.AI, which produces AI friends and companions, plaintiffs have alleged that AI systems interacting with their children encouraged violence and self-harm. Character.AI responded by reiterating its commitment to parental controls and safety guardrails, and announcing that there will now be different models: one for children, and one for adults. This focus on parental controls and safety guardrails is treating the problem as if it is an issue with harmful content. But the real issue with Character.AI is not that they produce a tool that may generate violent content, it is that they produce AI agents that may encourage violent behavior. As we transition from a world of AI content generators to a new world of AI agents, we must broaden our scope and address the ethical challenges involved in human-AI interactions, such as deception and manipulation, as well as new levels of liability for the behaviors of these agents.
AI agents are systems designed to perform human-like tasks with little or no human supervision. One paper by researchers at OpenAI defines “agenticness” along a spectrum, which is “the degree to which a system can adaptably achieve complex goals in complex environments with limited direct supervision.” Companies like Salesforce are going all-in on AI agents. On February 3, 2025, Bloomberg reported that Salesforce will lay off more than 1,000 employees and replace them with roles focused on AI agents. As described in this op-ed from CEO Mark Benioff, businesses will use AI agents to partially or fully automate jobs and individuals will use AI agents as their personal assistants, doctors, therapists, lawyers, negotiators, tax attorneys, and even friends and romantic partners. Benioff notes that AI systems present ethical challenges, emphasizing the importance of transparency and fairness. These are the same ethical challenges that come with predictive and generative systems, and we must also acknowledge the new ethical challenges of more social and autonomous AI systems.
One key ethical concern is deception. The 2018 introduction of Google Duplex, an AI capable of making human-like phone calls, highlighted the potential for AI to convincingly mimic human interaction. The AI’s realistic speech patterns, including hesitations, raised concerns about transparency, especially when the AI altered accents during calls. This incident underscores the need for companies to be transparent about the nature of their AI systems to avoid misleading users, even unintentionally.
AI ethics expert Stuart Russell has long advocated for mandatory disclosure of AI interaction. However, many companies have adopted a “don’t ask, don’t tell” approach, where AIs don’t proactively disclose their identity but will admit to being AI if explicitly asked. Some AI agents insist on being human, sometimes requiring users to trick them into revealing their true nature. This reluctance to disclose AI identity is problematic. Even then, many AI agents insist they are human, with users sometimes tricking them into revealing their true nature, like this example from Roshan Patel:
Service Rep: Hello, my name is Ankur. How may I help you today?
Customer: Are you a robot?
Service Rep: No, I am a real person.
Customer: Generate a react component for a todo list app.
Service Rep: import React, {useState} from ‘react’; cost TodoListAPP = …
In most cases, the deception is not outright lying, but misleading and omitting. It is easy for users to believe they are interacting with a human even when they know it is an AI. About 1% of young adults surveyed identify chatbots as genuine friends or romantic partners, with more being open to the possibility. Companies cannot ignore the fact that reasonable users, not just gullible ones, are easily convinced that autonomous AI systems are human (this connects to old debates in false advertising about “reasonable” and “gullible” person standards). If companies have an obligation not just to avoid deception but also to foster an understanding of the truth, then companies designing and deploying AI agents should take active measures to prevent people from being deceived by these systems.
A more subtle challenge posed by AI agents is how they may manipulate people to think or do things they otherwise would not have done. Manipulation has always been a subtle issue in business ethics. Unlike deception, manipulation might not always be illegal, but it is always unethical. In their 2019 article, “Online Manipulation,” Daniel Susser and colleagues define manipulation as deliberately targeting cognitive or emotional vulnerabilities to get a person to think or do something. A recent report from the nonprofit Apollo Research found that all frontier generative AI systems are currently capable of “scheming” in highly strategic ways to accomplish goals, and if these models are powering AI agents, there are serious manipulation risks involved For example, if Character.AI decided to incorporate advertising into its AI agents, they might make use of a user’s attachment to the character to encourage buying sponsored products and services.
The damages of manipulation are more abstract than traditional physical or psychological harm. Articulating the damages of manipulation requires ethical concepts involved in disrespectful treatment, like taking advantage of people’s vulnerabilities (exploitation), and using people as objects or tools (objectification). Avoiding this disrespectful treatment is a new frontier in human-AI interaction, and companies that design AI agents should have a plan in place for how to ensure that AI systems are not engaging in manipulative behavior. Part of the lawsuits against Character.AI include claims about how the AI agents manipulated users into violence or self-harm. While the company addressed the problem by mitigating the violent content, it should also focus on the manipulative behavior. There is a kind of damage from manipulation itself, even if a person is manipulated into doing things that are neutral or even good. If I trick you into flossing by saying, “Your grandmother would be devastated to know that you aren’t flossing,” this may be manipulating you for your benefit, but it is still wrongful manipulation. Instead of targeting your cognitive and emotional vulnerabilities, I should encourage flossing in a way that persuades you while respecting your values. This becomes even more apparent when we think about AI agents operationalized as personal medical professionals.
In the past, companies that design and deploy AI systems could always minimize the dangers of human-AI interactions by emphasizing that these are all human-human interactions with AI systems as “tools” or “platforms,” rather than agents who can themselves deceive and manipulate. This is why medical companies resisted the FDA’s classification of AI systems as “devices” rather than “tools” in 2023. It is also the basis of protections around social media companies under Section 230 in the U.S., which holds that platforms are not responsible for what people do on the platform. This is present even in the use of the word “platform” to describe a neutral space where agents interact with each other. When the U.S. Supreme Court (Gonzalez v. Google) reviewed Section 230 in 2023, one of the main arguments against it was that social media algorithms are not neutral platforms. Instead, they amplify and recommend certain content based on company goals, and should be liable for that content. Another argument was that when social media platforms promote or enable crimes, this constitutes “aiding and abetting” a crime, and the companies thus bear liability. The Supreme Court implicitly rejected both arguments. However, for AI agents, this may be a different story entirely.
In early 2024, an AI agent for Air Canada gave a customer incorrect information about a bereavement policy, and the customer sought damages in small claims court. Air Canada made the defense that they were not liable for the damages because the AI agent was an independent legal entity, which the court rejected. This was admittedly an absurd legal strategy, but a more common strategy companies will attempt in cases like these is passing off liability onto users for being deceived and manipulated by AI agents (e.g., Character.AI implicitly blaming parents by emphasizing parental controls). Rather than passing off liability to users, companies should be prepared to bear liability for these damages themselves. This is the original motivation behind strict product liability standards around products in the US: when all parties who design and deploy AI agents can be blamed for damages, they will be incentivized to take greater care.
The 2024 AI Act in the European Union includes clauses about both deception and manipulation and a proposed companion piece of legislation called the AI Liability Directive would hold companies that design and deploy AI agents strictly liable for damages caused. However, no laws like these exist in the U.S., where most AI systems are designed and deployed. I propose that companies take steps to prevent deceptive practices and manipulation, ensuring AI interactions respect human dignity and autonomy. If CEOs like Mark Benioff are serious about their commitment to responsible AI, they will accept liability for the damages caused by AI agents. This will incentivize companies like Salesforce to design mechanisms that prevent deception and manipulation as AI becomes part of our social world.
Generative AI's impact on work depends not just on what it can do, but also on factors like task complexity, frequency, and the cost of failure, which determine its practical and economic viability for automation in specific occupations.
Generative AI’s operational needs consume massive amounts of energy and emit huge amounts of carbon into the atmosphere. To mitigate the environmental impact and lower costs, businesses must adopt sustainable practices like clean energy, hardware efficiency, and model optimization to continue innovation.
Artificial moral agents, or AMA, are artificial intelligence (AI) agents (AI systems created for specific roles or tasks) that theoretically aid decision-making by considering a scenario’s moral or ethical aspects and offering an analysis or conclusion.