Can AI Agents Be Trusted?

Harvard Business Review

26-05-2025

Agentic AI has quickly become one of the most active areas of artificial intelligence development. AI agents are a level of programming on top of large language models (LLMs) that allow them to work towards specific goals. This extra layer of software can collect data, make decisions, take action, and adapt its behavior based on results. Agents can interact with other systems, apply reasoning, and work according to priorities and rules set by you as the principal.
Companies such as Salesforce have already deployed agents that can independently handle customer queries in a wide range of industries and applications, for example, and recognize when human intervention is required.
But perhaps the most exciting future for agentic AI will come in the form of personal agents, which can take self-directed action on your behalf. These agents will act as your personal assistant, handling calendar management, performing directed research and analysis, finding, negotiating for, and purchasing goods and services, curating content and taking over basic communications, learning and optimizing themselves along the way.
The idea of personal AI agents goes back decades, but the technology finally appears ready for prime-time. Already, leading companies are offering prototype personal AI agents to their customers, suppliers, and other stakeholders, raising challenging business and technical questions. Most pointedly: Can AI agents be trusted to act in our best interests? Will they work exclusively for us, or will their loyalty be split between users, developers, advertisers, and service providers? And how will be know?
The answers to these questions will determine whether and how quickly users embrace personal AI agents, and if their widespread deployment will enhance or damage business relationships and brand value.
What Could Go Wrong?
Think of a personal AI agent as someone you might hire as an employee, contractor or other real-world agent. Before delegating responsibility, you need to know if a person or business is reliable, honest, capable, and required by law to look out for you. For a human agent with the ability to commit your financial and other resources, for example, you would almost certainly conduct a background check, take out insurance, and, in some cases, require them to post a bond.
Depending on the duties of your personal AI agents, digital versions of these and other controls will be essential. That's because the risks of bad employees and contractors apply to personal AI agents, too. Indeed, given the potential scope and speed of agentic AI, users will need to be even more confident that their personal AI agents are trustworthy before turning over the keys to their most valuable assets. The most serious risks that must be addressed include:
Vulnerability to Criminals
A worst-case scenario is that personal AI agents could be programmed (or reprogrammed by hackers) to work against you, analogous to an identity thief or criminal employee embezzling funds.
It's too early for widespread reports of hijacked personal AI agents, but the U.S. National Institute of Standards and Technology and private Internet security firms have been conducting regular tests of leading LLMs and their agent technology for potential security flaws. These simulated hacks reveal that even today's most secure models can be easily tricked into performing malicious activities, including exposing passwords, sending phishing emails, and revealing proprietary software.
Retail Manipulation by Marketers and Paid Influencers
In retail, personal AI agents could be intentionally designed with biased marketing preferences to steer purchases towards those who develop them or their business partners.
Consider online shopping. Already, it's deluged by misleading advertising and paid promotion—much of which isn't disclosed. Consumer marketers have strong incentives to keep AI agents from shopping in a truly independent environment. 'Free' agents may steer business towards certain brands or retailers; worse, programmed bias in recommendations and purchases may be invisible to users.
Just as humans can be tricked into buying and selling from those who manipulate information unfairly or even illegally, AI agents may fall victim to similar abuse through software deployed by marketers to influence or even alter the LLMs that personal AI agents rely on. You believe your agent is finding you the best deal, but its analysis, decision-making and learning may be subtly or not-so-subtly altered by modifications to the inputs and reasoning it uses.
Preference for Sponsors and Advertisers
Manipulation can also include special preference for certain kinds of content or viewpoints. For instance, in news, entertainment, and social media, personal AI agents could be slanted to prioritize digital content or promote a service provider's sponsor instead of giving users the information that best meets their needs or preferences.
This is especially likely if the deployment of personal AI agents follows the approach of existing digital services, where users are given free or subsidized access to content, leaving platform operators to make their money from advertising, product placement, and other indirect sources linked to the content. As in the old days of ad-supported radio and television, that business model strongly aligns the interests of service providers not with those of their users but with their sponsors, leading to direct and indirect influence on content to best reflect the interests of advertisers and their brands.
Consider music service Spotify, which recently added a feature that allows subscribers to listen to music curated by an automated DJ, 'a personalized AI guide that knows you and your music taste so well that it can choose what to play for you.' Spotify also allows artists to have their work promoted in some user recommendation algorithms in exchange for a reduction in royalties, a system it refers to as 'Discovery Mode.' For now, Spotify confirmed that its AI DJ does not operate in conjunction with Discover Mode.
Susceptibility to Misinformation
Personal AI agent decision-making could be skewed intentionally or unintentionally by misinformation, a problem human principals and agents alike already face with today. This is perhaps the most general but also the most significant risk. Personal AI agents, for example, may be fooled, as are humans, by faked videos, which in some cases are used to blackmail or extort victims.
Examples of LLMs relying on erroneous or intentionally false information in response to user queries—in some cases giving dangerous health recommendations —have been regularly reported since the first release of ChatGPT and other early AI applications. Some courts have already held developers responsible when AI chatbots give incorrect answers or advice: For example, the case of an Air Canada passenger who was promised a discount that wasn't actually available.
Since the purveyors of false information have different objectives, including political, criminal, financial, or just plain maliciousness, it's difficult to gauge the risk that personal AI agents will inadvertently rely on such data in making consequential choices for their users.
Bringing Together Legal, Market, and Technical Solutions
One way to keep AI agents honest, just as with their human counterparts, is careful supervision, auditing, and limiting autonomy by establishing levels of approval based on the potential scale and cost of delegated decisions. Implementing such complex oversight over AI agents, however, would largely defeat the time-saving benefits of authorizing them to act on our behalf in the first place.
Instead, we believe the need for tedious micromanagement of AI agents by their users can be minimized by applying a combination of public and private regulation, insurance, and specialized hardware and software. Here are three key steps to ensuring trustworthy personal AI agents, some of which is already in development:
1. Treat AI Agents as Fiduciaries
Attorneys, legal guardians, trustees, financial advisors, board members, and other agents who manage the property or money of their clients are held to an enhanced duty of care, making them what is known as fiduciaries. Depending on the context, the legal responsibilities of a fiduciary vis-à-vis the client typically include obedience, loyalty, disclosure, confidentiality, accountability, and reasonable care and diligence in managing the client's affairs.
As a baseline, legal systems must ensure AI agents and any other software with the capability to make consequential decisions are treated as fiduciaries, with appropriate public and private enforcement mechanisms for breaches including failure to disclose potential conflicts of interest or failing to operate independently of paid influencers. Already, some legal scholars argue that existing precedent would treat personal AI agents as fiduciaries. If not, this may be a rare area of bi-partisan consensus on the need for regulation, with the leading developers of agentic AI technology themselves calling for legislation.
In the U.S., some fiduciaries are closely regulated by public agencies including the Securities and Exchange Commission and the Department of Labor, which oversee licensing, reporting, and disciplinary processes. Private self-regulatory bodies, such as bar associations, the Certified Financial Planner Board, and the National Association of Realtors can also act directly or indirectly to enforce fiduciary duties. Similar mechanisms, perhaps overseen by a new organization created by AI developers and corporate users, will need to monitor personal AI agents.
2. Encourage Market Enforcement of AI Agent Independence
Business leaders who will benefit from offering personal AI agents to their stakeholders should work together with service providers, private regulators, and entrepreneurs to promote trust and safety for agentic AI technology. This includes offering and including insurance in the deployment of personal AI agents.
For example, as retail and banking applications have exploded in use, a fast-growing, multi-billion dollar industry of identity theft protection quickly evolved to protect users against the unauthorized use of digital information by financial fiduciaries. Insurers in particular have strong incentives to police the practices of data managers and lobby for stronger laws or to engage private enforcement tools, including class action lawsuits, when appropriate.
Other service providers who already help users manage their online relationships with fiduciaries could expand their business to cover personal AI agents. Credit bureaus, for example, not only oversee a wide range of transactions and provide alerts based on user-defined criteria, they also provide consumers the ability to freeze their financial history so that criminals and other unauthorized users cannot open new lines of credit or manage credit history without explicit permission. (Since 2018, some of these tools must be offered free of charge to consumers in the U.S.)
Likewise, those deploying personal AI agents should encourage insurers and other service providers to give users the ability to monitor, control, and audit the behavior of their agents, independent of whoever creates and operates the software itself. AI 'credit bureaus' could offer tools to limit the autonomy of AI agents at user-defined levels, including the number or scale of consequential decisions the agent can make during a certain period of time.
3. Keep Decisions Local
Careful design and implementation of agentic AI technology can head off many trust-related issues before they arise. One effective way to deter commercial or criminal manipulation of personal AI agents is to restrict their ability to disclose personal data. Several device and operating system developers, including Google and Microsoft, are working on agentic AI tools that keep all sensitive data and decision-making performed by agents localized to the user's phone, tablet, or personal computer. This both limits the opportunity for outsiders to interfere with the agent and reduces the risk that sensitive data could be hijacked and used by rogue software posing as an authorized agent.
Apple Intelligence, Apple's AI architecture, will likewise limit most agent activity to a user's device. When more computing power is required, the company will use what it calls Private Cloud Compute (PCC), which can access larger LLMs and processing resources using Apple hardware and strong encryption. When using PCC, the company says, personal data will not be stored. The company has also committed to allowing independent privacy and security researchers to verify the integrity of the system at any time.
To ensure a rapid rollout of personal AI agents, all companies offering personal AI agents to their stakeholders should consider similar features, including strict localization of individual user data, strong encryption both for internal and external processing, and trustworthy business partners. Verifiable transparency of the agent's behavior, and full disclosure of sponsorships, paid promotions, and advertising interactions with personal AI agents are also essential.
Technical solutions like these are not foolproof, of course, but greatly limit the number of potential points of failure, reducing the risk that fiduciary responsibilities will not be fulfilled.
Getting Started
Agentic AI technology holds tremendous promise for making life easier and better, not only for enterprises but for individuals as well. Still, users will not embrace AI agents unless they are confident the technology can be trusted, that there is both public and private oversight of agent behavior, and appropriate monitoring, reporting, and customization tools that are independent of the developers of the agents themselves.
Getting it right, as with any fiduciary relationship, will require a clear assignment of legal rights and responsibilities, supported by a robust market for insurance and other forms of third-party protection and enforcement tools. Industry groups, technology developers, consumer services companies, entrepreneurs, users, consumer advocates, and lawmakers must come together now to accelerate adoption of this key technology.