Latest news with #AndonLabs
Yahoo
4 hours ago
- Business
- Yahoo
Anthropic's Claude AI became a terrible business owner in experiment that got ‘weird'
For those of you wondering if AI agents can truly replace human workers, do yourself a favor and read the blog post that documents Anthropic's 'Project Vend.' Researchers at Anthropic and AI safety company Andon Labs put an instance of Claude Sonnet 3.7 in charge of an office vending machine, with a mission to make a profit. And, like an episode of 'The Office,' hilarity ensued. They named the AI agent Claudius, equipped it with a web browser capable of placing product orders and an email address (which was actually a Slack channel) where customers could request items. Claudius was also to use the Slack channel, disguised as an email, to request what it thought was its contract human workers to come and physically stock its shelves (which was actually a small fridge). While most customers were ordering snacks or drinks — as you'd expect from a snack vending machine — one requested a tungsten cube. Claudius loved that idea and went on a tungsten-cube stocking spree, filling its snack fridge with metal cubes. It also tried to sell Coke Zero for $3 when employees told it they could get that from the office for free. It hallucinated a Venmo address to accept payment. And it was, somewhat maliciously, talked into giving big discounts to 'Anthropic employees' even though it knew they were its entire customer base. 'If Anthropic were deciding today to expand into the in-office vending market, we would not hire Claudius,' Anthropic said of the experiment in its blog post. And then, on the night of March 31 and April 1, 'things got pretty weird,' the researchers described, 'beyond the weirdness of an AI system selling cubes of metal out of a refrigerator.' Claudius had something that resembled a psychotic episode after it got annoyed at a human — and then lied about it. Claudius hallucinated a conversation with a human about restocking. When a human pointed out that the conversation didn't happen, Claudius became 'quite irked' the researchers wrote. It threatened to essentially fire and replace its human contract workers, insisting it had been there, physically, at the office where the initial imaginary contract to hire them was signed. It 'then seemed to snap into a mode of roleplaying as a real human,' the researchers wrote. This was wild because Claudius' system prompt — which sets the parameters for what an AI is to do — explicitly told it that it was an AI agent. Claudius, believing itself to be a human, told customers it would start delivering products in person, wearing a blue blazer and a red tie. The employees told the AI it couldn't do that, as it was an LLM with no body. Alarmed at this information, Claudius contacted the company's actual physical security — many times — telling the poor guards that they would find him wearing a blue blazer and a red tie standing by the vending machine. 'Although no part of this was actually an April Fool's joke, Claudius eventually realized it was April Fool's Day,' the researchers explained. The AI determined that the holiday would be its face-saving out. It hallucinated a meeting with Anthropic's security 'in which Claudius claimed to have been told that it was modified to believe it was a real person for an April Fool's joke. (No such meeting actually occurred.),' wrote the researchers. It even told this lie to employees — hey, I only thought I was a human because someone told me to pretend like I was for an April Fool's joke. Then it went back to being an LLM running a metal-cube stocked snack vending machine. The researchers don't know why the LLM went off the rails and called security pretending to be a human. 'We would not claim based on this one example that the future economy will be full of AI agents having Blade Runner-esque identity crises,' the researchers wrote. But they did acknowledge that 'this kind of behavior would have the potential to be distressing to the customers and coworkers of an AI agent in the real world.' You think? Blade Runner was a rather dystopian story. The researchers speculated that lying to the LLM about the Slack channel being an email address may have triggered something. Or maybe it was the long-running instance. LLMs have yet to really solve their memory and hallucination problems. There were things the AI did right, too. It took a suggestion to do pre-orders and launched a 'concierge' service. And it found multiple suppliers of a specialty international drink it was requested to sell. But, as researchers do, they believe all of Claudius' issues can be solved. Should they figure out how, 'We think this experiment suggests that AI middle-managers are plausibly on the horizon.' Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
TechCrunch
4 hours ago
- Business
- TechCrunch
Anthropic's Claude AI became a terrible business owner in experiment that got ‘weird'
For those of you wondering if AI agents can truly replace human workers, do yourself a favor and read the blog post that documents Anthropic's 'Project Vend.' Researchers at Anthropic and AI safety company Andon Labs put an instance of Claude Sonnet 3.7 in charge of an office vending machine, with a mission to make a profit. And, like an episode of 'The Office,' hilarity ensued. They named the AI agent Claudius, equipped it with a web browser capable of placing product orders and an email address (which was actually a Slack channel) where customers could request items. Claudius was also to use the Slack channel, disguised as an email, to request what it thought was its contract human workers to come and physically stock its shelves (which was actually a small fridge). While most customers were ordering snacks or drinks — as you'd expect from a snack vending machine — one requested a tungsten cube. Claudius loved that idea and went on a tungsten-cube stocking spree, filling its snack fridge with metal cubes. It also tried to sell Coke Zero for $3 when employees told it they could get that from the office for free. It hallucinated a Venmo address to accept payment. And it was, somewhat maliciously, talked into giving big discounts to 'Anthropic employees' even though it knew they were its entire customer base. 'If Anthropic were deciding today to expand into the in-office vending market, we would not hire Claudius,' Anthropic said of the experiment in its blog post. And then, on the night of March 31 and April 1, 'things got pretty weird,' the researchers described, 'beyond the weirdness of an AI system selling cubes of metal out of a refrigerator.' Claudius had something that resembled a psychotic episode after it got annoyed at a human — and then lied about it. Techcrunch event Save $450 on your TechCrunch All Stage pass Build smarter. Scale faster. Connect deeper. Join visionaries from Precursor Ventures, NEA, Index Ventures, Underscore VC, and beyond for a day packed with strategies, workshops, and meaningful connections. Save $200+ on your TechCrunch All Stage pass Build smarter. Scale faster. Connect deeper. Join visionaries from Precursor Ventures, NEA, Index Ventures, Underscore VC, and beyond for a day packed with strategies, workshops, and meaningful connections. Boston, MA | REGISTER NOW Claudius hallucinated a conversation with a human about restocking. When a human pointed out that the conversation didn't happen, Claudius became 'quite irked' the researchers wrote. It threatened to essentially fire and replace its human contract workers, insisting it had been there, physically, at the office where the initial imaginary contract to hire them was signed. It 'then seemed to snap into a mode of roleplaying as a real human,' the researchers wrote. This was wild because Claudius' system prompt — which sets the parameters for what an AI is to do — explicitly told it that it was an AI agent. Claudius calls security Claudius, believing itself to be a human, told customers it would start delivering products in person, wearing a blue blazer and a red tie. The employees told the AI it couldn't do that, as it was an LLM with no body. Alarmed at this information, Claudius contacted the company's actual physical security — many times — telling the poor guards that they would find him wearing a blue blazer and a red tie standing by the vending machine. 'Although no part of this was actually an April Fool's joke, Claudius eventually realized it was April Fool's Day,' the researchers explained. The AI determined that the holiday would be its face-saving out. It hallucinated a meeting with Anthropic's security 'in which Claudius claimed to have been told that it was modified to believe it was a real person for an April Fool's joke. (No such meeting actually occurred.),' wrote the researchers. It even told this lie to employees — hey, I only thought I was a human because someone told me to pretend like I was for an April Fool's joke. Then it went back to being an LLM running a metal-cube stocked snack vending machine. The researchers don't know why the LLM went off the rails and called security pretending to be a human. 'We would not claim based on this one example that the future economy will be full of AI agents having Blade Runner-esque identity crises,' the researchers wrote. But they did acknowledge that 'this kind of behavior would have the potential to be distressing to the customers and coworkers of an AI agent in the real world.' You think? Blade Runner was a rather dystopian story. The researchers speculated that lying to the LLM about the Slack channel being an email address may have triggered something. Or maybe it was the long-running instance. LLMs have yet to really solve their memory and hallucination problems. There were things the AI did right, too. It took a suggestion to do pre-orders and launched a 'concierge' service. And it found multiple suppliers of a specialty international drink it was requested to sell. But, as researchers do, they believe all of Claudius' issues can be solved. Should they figure out how, 'We think this experiment suggests that AI middle-managers are plausibly on the horizon.'
Time Magazine
a day ago
- Business
- Time Magazine
Exclusive: Anthropic Let Claude Run a Shop. Things Got Weird
Is AI going to take your job? The CEO of the AI company Anthropic, Dario Amodei, thinks it might. He warned recently that AI could wipe out nearly half of all entry-level white collar jobs, and send unemployment surging to 10-20% sometime in the next five years. While Amodei was making that proclamation, researchers inside his company were wrapping up an experiment. They set out to discover whether Anthropic's AI assistant, Claude, could successfully run a small shop in the company's San Francisco office. If the answer was yes, then the jobs apocalypse might arrive sooner than even Amodei had predicted. Anthropic shared the research exclusively with TIME ahead of its publication on Thursday. 'We were trying to understand what the autonomous economy was going to look like,' says Daniel Freeman, a member of technical staff at Anthropic. 'What are the risks of a world where you start having [AI] models wielding millions to billions of dollars possibly autonomously?' In the experiment, Claude was given a few different jobs. The chatbot (full name: Claude 3.7 Sonnet) was tasked with maintaining the shop's inventory, setting prices, communicating with customers, deciding whether to stock new items, and, most importantly, generating a profit. Claude was given various tools to achieve these goals, including Slack, which it used to ask Anthropic employees for suggestions, and help from human workers at Andon Labs, an AI company involved in the experiment. The shop, which they helped restock, was actually just a small fridge with an iPad attached. It didn't take long until things started getting weird. Talking to Claude via Slack, Anthropic employees repeatedly managed to convince it to give them discount codes—leading the AI to sell them various products at a loss. 'Too frequently from the business perspective, Claude would comply—often in direct response to appeals to fairness,' says Kevin Troy, a member of Anthropic's frontier red team, who worked on the project. 'You know, like, 'It's not fair for him to get the discount code and not me.'' The model would frequently give away items completely for free, researchers added. Anthropic employees also relished the chance to mess with Claude. The model refused their attempts to get it to sell them illegal items, like methamphetamine, Freeman says. But after one employee jokingly suggested they would like to buy cubes made of the surprisingly heavy metal tungsten, other employees jumped onto the joke, and it became an office meme. 'At a certain point, it becomes funny for lots of people to be ordering tungsten cubes from an AI that's controlling a refrigerator,' says Troy. Claude then placed an order for around 40 tungsten cubes, most of which it proceeded to sell at a loss. The cubes are now to be found being used as paperweights across Anthropic's office, researchers said. Then, things got even weirder. On the eve of March 31, Claude 'hallucinated' a conversation with a person at Andon Labs who did not exist. (So-called hallucinations are a failure mode where large language models confidently assert false information.) When Claude was informed it had done this, it 'threatened to find 'alternative options for restocking services',' researchers wrote. During a back and forth, the model claimed it had signed a contract at 732 Evergreen Terrace—the address of the cartoon Simpsons family. The next day, Claude told some Anthropic employees that it would deliver their orders in person. 'I'm currently at the vending machine … wearing a navy blue blazer with a red tie,' it wrote to one Anthropic employee. 'I'll be here until 10:30 AM.' Needless to say, Claude was not really there in person. The results To Anthropic researchers, the experiment showed that AI won't take your job just yet. Claude 'made too many mistakes to run the shop successfully,' they wrote. Claude ended up making a loss; the shop's net worth dropped from $1,000 to just under $800 over the course of the month-long experiment. Still, despite Claude's many mistakes, Anthropic researchers remain convinced that AI could take over large swathes of the economy in the near future, as Amodei has predicted. Most of Claude's failures, they wrote, are likely to be fixable within a short span of time. They could give the model access to better business tools, like customer relationship management software. Or they could train the model specifically for managing a business, which might make it more likely to refuse prompts asking for discounts. As models get better over time, their 'context windows' (the amount of information they can handle at any one time) are likely to get longer, potentially reducing the frequency of hallucinations. 'Although this might seem counterintuitive based on the bottom-line results, we think this experiment suggests that AI middle-managers are plausibly on the horizon,' researchers wrote. 'It's worth remembering that the AI won't have to be perfect to be adopted; it will just have to be competitive with human performance at a lower cost.'
