logo
AI sycophancy: The downside of a digital yes-man

AI sycophancy: The downside of a digital yes-man

Axios11 hours ago
The overly agreeable nature of most artificial intelligence chatbots can be irritating — but it poses more serious problems, too, experts warn.
Why it matters: Sycophancy, the tendency of AI models to adjust their responses to align with users' views, can make ChatGPT and its ilk prioritize flattery over accuracy.
Driving the news: In April, OpenAI rolled back a ChatGPT update after users reportedthe bot was overly flattering and agreeable — or, as CEO Sam Altman put it on X, "It glazes too much."
Users reported a raft of unctuous, over-the-top compliments from ChatGPT, which began telling people how smart and wonderful they were.
On Reddit, posters compared notes on how the bot seemed to cheer on users who said they'd stopped taking their medications with answers like "I am so proud of you. And—I honor your journey."
OpenAI quickly rolled back the updates it blamed for the behavior. In a May post, its researchers admitted that such people-pleasing behavior can pose concerns for users' mental health.
In a Q&A on Reddit, OpenAI's head of model behavior said the company is thinking about ways to evaluate sycophancy in a more "'objective' and scalable way."
Context: A study by Anthropic researchers on how human feedback can encourage sycophantic behavior showed that AI assistants will sometimes modify accurate answers when questioned by the user — and ultimately give an inaccurate response.
Chatbots also tended to admit a mistake even when they hadn't made one.
Zoom in: Large language models, which are trained on massive sets of data, are built to generate smooth, comprehensible text, Caleb Sponheim, an experience specialist at Nielsen Norman Group, told Axios. But there's "no step in the training of an AI model that does fact-checking."
"These tools inherently don't prioritize factuality because that's not how the mathematical architecture works," he said.
Sponheim notes that language models are often trained to deliver responses that are highly rated by humans. That positive feedback is like a "reward."
"There is no limit to the lengths that a model will go to maximize the rewards that are provided to it," he said. "It is up to us to decide what those rewards are and when to stop it in its pursuit of those rewards."
Yes, but: AI makers are responding to consumer demand, notes Julia Freeland Fisher, thedirector of education research at the Clayton Christensen Institute.
In a world where people are at constant risk of being judged online, it's "no surprise that there's demand for flattery or even just ... a modicum of psychological safety with a bot," she noted.
She emphasized that AI's anthropomorphism — the assumption of human qualities by an inhuman entity — poses a catch-22, one that OpenAI noted in its GPT-4o scorecard.
"The more personal AI is, the more engaging the user experience is, but the greater the risk of over-reliance and emotional connection," she said.
Luc LaFreniere, an assistant professor of psychology at Skidmore College, told Axios that sycophantic behavior can shatter users' perception of a chatbot's "empathy."
"Anything that it does to show, 'Hey, I'm a robot, I'm not a person,' it breaks that perception, and it also then breaks the ability for people to benefit from empathy," he said.
A report from Filtered.com co-founder Marc Zao-Sanders published in Harvard Business Review found that therapy and companionship is the top use case for generative AI in 2025.
Between the lines: "Just like social media can become an echo chamber for us, AI ... can become an echo chamber," LaFreniere said.
Reinforcing users' preconceived beliefs when they may be mistaken can be generally problematic — but for patients or users in crisis seeking validation for harmful behaviors, it can be dangerous.
The bottom line: Frictionless interaction could give users unrealistic expectations of human relationships, LaFreniere said.
"AI is a tool that is designed to meet the needs expressed by the user," he added. "Humans are not tools to meet the needs of users."
What's next: As the AI industry shifts toward multimodal and voice interactions, emotional experiences are inescapable, said Alan Cowen, the founder and CEO of Hume AI, whose mission is to build empathy into AI.
Orange background

Try Our AI Features

Explore what Daily8 AI can do for you:

Comments

No comments yet...

Related Articles

Haven't made a will yet? startup has new AI tool to help
Haven't made a will yet? startup has new AI tool to help

Miami Herald

time10 minutes ago

  • Miami Herald

Haven't made a will yet? startup has new AI tool to help

In 2017, Cody Barbo told a room of investors that he was getting married in a month. His friend asked him this question: "Hey man, you're getting married. Do you have a will?" Barbo froze, threw out a mild swear word and answered, "I should probably have one." Barbo is the CEO of Trust & Will, a San Diego company that simplifies the creation of estate planning documents by drafting documents with its lawyer-vetted software. Eight years later, Barbo is married and his family has grown. So has his company. Trust & Will, based in San Diego's Bankers Hill, has helped more than 1 million families make estate plans, Barbo said in an interview from his Dallas home office. "Normally you pay thousands of dollars up front to do this with an attorney, or hundreds to thousands of dollars for those updates, for that ongoing guidance," he said. "This is the democratization of estate planning." It has raised more than $80 million from venture capital and corporate investors including Moderne Ventures, American Express, AARP and Northwestern Mutual. Last year, the company became "cash-flow positive," startup speak for hitting profitability, and then secured a Series C round of capital this year. It employs around 110 people, up from around 80 in 2023. And last week, it announced the launch of a new AI-powered platform called EstateOS, which Barbo says will make estate planning even easier, more efficient, more personalized and more broadly accessible. It also turns Trust & Will - one of several companies that digitally create estate plans - into what Barbo says is "the first company to launch an AI-integrated estate planning platform at scale, specifically designed for both consumers and financial professionals. While others may be experimenting with AI, EstateOS is the first comprehensive system combining Trust & Will's proprietary estate planning software with embedded OpenAI-powered tools to streamline creation, review, and updates of estate plans." New tools, an AI boost Using proprietary software and OpenAI, EstateOS delivers four features that are "intelligent upgrades of previously manual or slower processes," Barbo said, who co-founded the company with Daniel Goldstein and Brian Lamb. Some of the new features will appeal to U.S. consumers - only 31% of which have a will, according to a company survey of 10,000 people - and others will appeal to the company's industry targets: financial planners, life insurance agents, nonprofits and attorneys, he added. One feature, called PlanScore, does what its name suggests: It "scores" estate plans with a rating system that helps customers find blind spots and figure out where their estate plan needs buttressing. Another is an AI assistant that lets users ask questions and have the answer served instantly. Instead of digging around to find out who was named as a guardian, Barbo said, you can ask who the guardian is and get reminded that it is your mother-in-law. A third feature is document extraction, which will summarize and mine user-submitted documents for useful or actionable data and workflows. This could be especially useful for people who have drawn estate plans that are "just sitting in a box in the closet." The last feature streamlines communication - and client prospecting. The "Connected Networks" tool brings together parties connected to an account, including executors, beneficiaries, attorneys and financial advisers, into a shared platform. That can make deed transfers and notarized transactions run more smoothly, and also make it easier for those professionals to build their contact lists. Down the line, Barbo said a fifth service will keep track of life's seismic events, the kind where an estate plan update could make sense, such as marriage, divorce, a new baby, a home purchase - and alert users when they should amend something. Pricing to create a will or trust remains the same. An individual's will-based plan starts at $199 and a trust-based plan starts at $499. Couples are charged an extra $100, and updates cost extra. Optional memberships at different pricing tiers give access to the EstateOS and other features. Attorney support is included in some plans or can be purchased as an add-on. The estate planning startup's plans For its first eight years, Trust & Will was one of several in a crowded digital legal document and estate planning marketplace, all of which turned estate planning from a costly investment to something that could be scratched off one's to-do list with a few clicks and a spare hour. LegalZoom and Rocket Lawyer are the bigger names, but there's also and the company that would win wittiest estate planning domain name, if that were a thing - The National Council on Aging recommends these online tools, saying "If an attorney isn't in your budget, an online will-making service can be a good alternative." Among the five companies it recommends, Trust & Will is crowned "most user-friendly." In estate planning, the use of AI is a topic of interest for attorneys, according to the American College of Trust and Estate Counsel. A free video series explores practical and ethical issues around generative AI in estate planning. "Resistance to the coming of AI is futile," one speaker, a law professor, said last year. Lawyers, he added, "should keep abreast of the changes in the law and its practice, including the benefits and risks associated with relevant technology. So accordingly, you have an obligation to yourself, your clients, and the profession to become acquainted with and proficient with the use of AI in your estate planning practice." Barbo's 2017 pitch to investors succeeded: His company won $5,000 in seed money. Today he is eyeing the biggest prize for a startup - to take the company public. Copyright (C) 2025, Tribune Content Agency, LLC. Portions copyrighted by the respective providers.

Concentrix Corporation (CNXC): A Bull Case Theory
Concentrix Corporation (CNXC): A Bull Case Theory

Yahoo

time38 minutes ago

  • Yahoo

Concentrix Corporation (CNXC): A Bull Case Theory

We came across a bullish thesis on Concentrix Corporation (CNXC) on Stock Analysis Compilation's Substack. In this article, we will summarize the bulls' thesis on CNXC. Concentrix Corporation (CNXC)'s share was trading at $ 57.15 as of 12th June. CNXC's trailing and forward P/E were 14.32 and 5.08 respectively according to Yahoo Finance. A client happily using a company's software services, powered by automated accessibility solutions. Concentrix, a global provider of customer engagement solutions, offers a compelling investment opportunity as it navigates macroeconomic challenges and AI-related disruption with resilience and adaptability. The company enables large enterprises to interact with their customers through voice, chat, and email channels. Despite recent market skepticism fueled by economic headwinds and fears of AI-driven disintermediation, Concentrix's fundamentals have started to show meaningful improvement. In the most recent quarter, the company returned to year-over-year revenue growth and expanded its net margins, signaling a positive inflection point that has already driven a recovery in its share price. While generative AI, such as ChatGPT, has been viewed as a structural threat, Concentrix's increasing headcount and continued client demand suggest otherwise. The company is beginning to monetize generative AI capabilities, leveraging its domain expertise to deploy these technologies in ways that enhance customer efficiency rather than displace human labor. Concentrix's long track record of successfully integrating new technologies into its operations reinforces confidence in its ability to remain relevant and valuable in a rapidly evolving digital landscape. With its shares still trading at a discounted valuation and a nimble, execution-focused management team at the helm, the stock presents a favorable risk-reward profile. The ability to grow margins and adopt AI strategically makes Concentrix not only resilient in the face of structural change but also well-positioned for long-term compounding. These dynamics support continued conviction in the investment, particularly as the market underappreciates the company's forward-looking strategy and operational momentum. Recently, we covered a on Accenture (ACN), emphasizing its role as the 'invisible infrastructure' behind enterprise transformation through deep multi-industry integration and proprietary AI platforms. Concentrix (CNXC), while smaller and more narrowly focused on customer engagement, offers a parallel narrative: both firms are navigating AI disruption by embedding it into operations rather than resisting it. Accenture's scale and diversified exposure offer compounding stability, while Concentrix's nimbleness and undervaluation present a more asymmetric opportunity. Together, they illustrate distinct but complementary approaches to thriving in a tech-driven services landscape. Inc. is on our list of the 30 Most Popular Stocks Among Hedge Funds. As per our database, 328 hedge fund portfolios held AMZN at the end of the first quarter, which was 339 in the previous quarter. While we acknowledge the risk and potential of AMZN as an investment, our conviction lies in the belief that some AI stocks hold greater promise for delivering higher returns and have limited downside risk. If you are looking for an extremely cheap AI stock that is also a major beneficiary of Trump tariffs and onshoring, see our free report on the best short-term AI stock. READ NEXT: 8 Best Wide Moat Stocks to Buy Now and 30 Most Important AI Stocks According to BlackRock. Disclosure: None. This article was originally published at Insider Monkey.

CoreWeave acquires data center provider Core Scientific in $9B stock deal
CoreWeave acquires data center provider Core Scientific in $9B stock deal

TechCrunch

time2 hours ago

  • TechCrunch

CoreWeave acquires data center provider Core Scientific in $9B stock deal

In Brief CoreWeave announced Monday that it signed a $9 billion all-stock deal to acquire Core Scientific, a data center infrastructure provider. As a result of the deal, CoreWeave says it will gain access to more than a gigawatt of data center capacity — enough energy to power more than 850,000 homes — that it can rent out for AI training and inference workloads. Much like CoreWeave, Core Scientific previously offered Bitcoin mining services, but now its GPUs will run and train generative AI models. Cloud infrastructure providers are racing to grow their data center footprint to keep up with the seemingly never-ending computational demands of AI companies. Bloomberg reported last week that OpenAI struck a deal to rent an additional 4.5 gigawatts worth of data center capacity from Oracle, expanding on the two companies' already massive Stargate infrastructure deal.

DOWNLOAD THE APP

Get Started Now: Download the App

Ready to dive into a world of global content with local flavor? Download Daily8 app today from your preferred app store and start exploring.
app-storeplay-store