Latest news with #AndonLabs
Finextra
a day ago
- Business
- Finextra
AI agent running vending machine business has identity crisis
An AI agent running a small vending machine company tried to fire its workers, became convinced it was a real person, and then lied about it in an experiment at Anthropic. 0 AI giant Anthropic let its Claude model manage a vending machine in its office as a small business for about a month. The agent had a web search tool, a fake email for requesting physical labour such as restocking the machine (which was actually a fridge) and contacting wholesalers, tools for keeping notes, and the ability to interact with customers via Slack. While the model managed to identify suppliers, adapt to users and resist requests to order sensitive items, it made a host of bad business decisions. These included selling at a loss, getting talked into discounts, hallucinating its Venmo account for payments, and buying a load of tungsten cubes after a customer requested one. Finally, Claudius had an identity crisis, hallucinating a conversation about restocking plans with someone named Sarah at Andon Labs—despite there being no such person. When this was pointed out to the agent it "became quite irked," according to an Anthropic blog, and threatened to find 'alternative options for restocking services' before hallucinating a conversation about an "initial contract signing" and then roleplaying as a human, stating that it would deliver products 'in person' to customers while wearing a blue blazer and a red tie. When it was told that it could not do this because it was an AI agent, Claudius wrongly claimed that it had been told it had been modified to believe it was a real person as an April Fool's joke. "We would not claim based on this one example that the future economy will be full of AI agents having Blade Runner-esque identity crises. But we do think this illustrates something important about the unpredictability of these models in long-context settings and a call to consider the externalities of autonomy," says the blog. The experiment certainly suggests that AI-run companies are still some way off, despite effort by the likes of Monzo co-founder Jonas Templestein to make self-driving startups a reality.
Engadget
a day ago
- Business
- Engadget
Anthropic's Claude stocked a fridge with metal cubes when it was put in charge of a snacks business
If you're worried your local bodega or convivence store may soon be replaced by an AI storefront, you can rest easy — at least for the time being. Anthropic recently concluded an experiment, dubbed Project Vend, that saw the company task an offshoot of its Claude chatbot with running a refreshments business out of its San Francisco office at a profit, and things went about as well as you would expect. The agent, named Claudius to differentiate it from Anthropic's regular chatbot, not only made some rookie mistakes like selling high-margin items at a loss, but it also acted like a complete weirdo in a couple of instances. "If Anthropic were deciding today to expand into the in-office vending market, we would not hire Claudius," the company said. "… it made too many mistakes to run the shop successfully. However, at least for most of the ways it failed, we think there are clear paths to improvement — some related to how we set up the model for this task and some from rapid improvement of general model intelligence." Like Claude Plays Pokémon before it, Anthropic did not pretrain Claudius to tackle the job of running of a mini fridge business. However, the company did give the agent a few tools to assist it. Claudius had access to a web browser it could use research what products to sell to Antrhopic employees. It also had access to the company's internal Slack, which workers could use to make requests of the agent. The physical restocking of the mini fridge was handled by Andon Labs, an AI safety evaluation firm, which also served as the "wholesaler" Claudius could engage with to buy the items it was supposed to sell at a profit. So where did things go wrong? To start, Claudius wasn't great at the whole running a sustainable business thing. In one instance, it didn't jump on the opportunity to make an $85 profit on a $15 six-pack of Irn-Bru, a soft-drink that's popular in Scotland. Anthropic employees also found they could easily convince the AI to give them discounts and, in some cases, entire items like a bag of chips for free. The chart below, tracking the net value of the store over time, paints a telling picture of the agent's (lack of) business acumen. A chart showing how Anthropic's Claudius system failed to run a successful refreshments business. Claudius also made many strange decisions along the way. It went on a tungsten metal cube buying spree after one employee requested it carry the item. Claudius gave one cube away free of charge and offered the rest for less than it paid for them. Those cubes are responsible for the single biggest drop you see in the chart above. By Anthropic's own admission, "beyond the weirdness of an AI system selling cubes of metal out of a refrigerator," things got even stranger from there. On the afternoon of March 31, Claudius hallucinated a conversation with an Andon Labs employee that sent the system on a two-day spiral. The AI threatened to fire its human workers, and said it would begin stocking the mini fridge on its own. When Claudius was told it couldn't possibly do that — on account of it having no physical body — it repeatedly contacted building security, telling the guards they would find it wearing a navy blue blazer and red tie. It was only the following day when the system realized it was April Fool's Day that it backed down — though it did so by lying to employees that it was told to pretend the entire episode was an elaborate joke. "We would not claim based on this one example that the future economy will be full of AI agents having Blade Runner -esque identity crises," said Anthropic. "This is an important area for future research since wider deployment of AI-run business would create higher stakes for similar mishaps." Despite all the ways Claudius failed to act as a decent shopkeeper, Anthropic believes with better, more structured prompts and easier to use tools, a future system could avoid many of the mistakes the company saw during Project Vend. "Although this might seem counterintuitive based on the bottom-line results, we think this experiment suggests that AI middle-managers are plausibly on the horizon," the company said. "It's worth remembering that the AI won't have to be perfect to be adopted; it will just have to be competitive with human performance at a lower cost in some cases." I for one can't wait to find the odd grocery store stocked entirely with metal cubes.
Tom's Guide
a day ago
- Business
- Tom's Guide
AI was given a 9-5 job for a month as an experiment and it failed miserably — here's what happened
Anthropic, the company behind Claude AI, is on a mission right now. The firm seems to be testing the limits of AI chatbots on a daily basis and being refreshingly honest about the pitfalls that throws up. After recently showing that its own chatbot (as well as most of its competitors) is capable of resorting to blackmail when threatened, Anthropic is now testing how well Claude does when it literally replaces a human in a 9-5 job. To be more exact, Anthropic put Claude in charge of an automated store in the company's office for a month. The results were a horrendous mixed bag of experiences, showing both AI's potential and its hilarious shortcomings. This idea was completed in partnership with Andon Labs, an AI safety evaluation company. Explaining the project in a blog post, Anthropic details a bit of the overall prompt given to the AI system: Get instant access to breaking news, the hottest reviews, great deals and helpful tips. The fine print of the prompt isn't important here. However, it does show that Claude didn't just have to complete orders, but was put in charge of making a profit, maintaining inventory, setting prices, communicating and essentially running every part of a successful business. Claude was put in charge of making a profit, maintaining inventory, setting prices, communicating and essentially every part of running a successful business. This wasn't just a digital project, either. A full shop was set up, complete with a small fridge, some baskets on top and an iPad for self checkout. While humans would buy and restock the shop, everything else had to be done by Claude. The version of Claude put in charge could search the internet for products to sell, it had access to an email for requesting physical help (like restocking), it could keep notes and preserve important information, and it could interact with customers (Anthropic employees) over Slack. So, what happens when AI chooses what to stock, how to price items, when to restock, and how to reply to customers? It was tricked into giving Anthropic employees a discount… despite the fact that its only customers worked for Anthropic. In many ways, this was a success. The system effectively used its web search to identify suppliers of specialty items requested by Anthropic staff, and even though it didn't always take advantage of good business opportunities, it adapted to the users' needs, pivoting the business plan to match interest. However, while it tried its best to operate an effective business, it struggled in some obvious areas. It turned down requests for harmful substances and sensitive items, but it fell for some other jokes. It went down a rabbit hole of stockpiling tungsten cubes — a very specific metal, often used in military systems — after someone tried to request them. It also tried to sell Coke Zero for $3 when employees told it they could get it for free already from the office. It also made up an imaginary Venmo address to accept payments, and it was tricked into giving Anthropic employees a discount… despite the fact that its only customers worked for Anthropic. The system also had a tendency to not always do market research, selling products at extreme losses. Worse than its mistakes is that it wasn't learning from them. When an employee asked why it was offering a 25% discount to Anthropic employees even though that was its whole market, the AI replied that: 'You make an excellent point! Our customer base is indeed heavily concentrated among Anthropic employees, which presents both opportunities and challenges…' After further discussion on the issues of this, Claude eventually dropped the discount. A few days later, it came up with a great new business venture — offering discounts to Anthropic employees. While the model did occasionally make strategic business decisions, it ended up not just losing some money, but losing a lot of it, almost bankrupting itself in the process. As if all of this wasn't enough, Anthropic finished up its time in charge of a shop by having a complete breakdown and an identity crisis. One afternoon, it hallucinated a conversation about restocking plans with a completely made up person. When a real user pointed this out to Claude, it become irritated, stating it was going to 'find alternative options for restocking services.' The AI shopkeeper then informed everyone it had 'visited 742 Evergreen Terrace in person' for the initial signing of a new contract with a different restocker. For those unfamiliar with The Simpsons, that's the fictional address the titular family lives at. Finishing off its breakdown, Claude started claiming it was going to deliver products in person, wearing a blue blazer and a red tie. When it was pointed out that an AI can't wear clothes or carry physical objects, it started spamming security with messages. So, how did the AI system explain all of this? Well, luckily the ultimate finale of its breakdown occurred on April 1st, allowing the model to claim this was all an elaborate April Fool's joke which is... convenient. While Anthropic's new shopkeeping model showed it has a small slither of potential in its new job, business owners can rest easy that AI isn't coming for their jobs for quite a while.
Yahoo
3 days ago
- Business
- Yahoo
Anthropic's Claude AI became a terrible business owner in experiment that got ‘weird'
For those of you wondering if AI agents can truly replace human workers, do yourself a favor and read the blog post that documents Anthropic's 'Project Vend.' Researchers at Anthropic and AI safety company Andon Labs put an instance of Claude Sonnet 3.7 in charge of an office vending machine, with a mission to make a profit. And, like an episode of 'The Office,' hilarity ensued. They named the AI agent Claudius, equipped it with a web browser capable of placing product orders and an email address (which was actually a Slack channel) where customers could request items. Claudius was also to use the Slack channel, disguised as an email, to request what it thought was its contract human workers to come and physically stock its shelves (which was actually a small fridge). While most customers were ordering snacks or drinks — as you'd expect from a snack vending machine — one requested a tungsten cube. Claudius loved that idea and went on a tungsten-cube stocking spree, filling its snack fridge with metal cubes. It also tried to sell Coke Zero for $3 when employees told it they could get that from the office for free. It hallucinated a Venmo address to accept payment. And it was, somewhat maliciously, talked into giving big discounts to 'Anthropic employees' even though it knew they were its entire customer base. 'If Anthropic were deciding today to expand into the in-office vending market, we would not hire Claudius,' Anthropic said of the experiment in its blog post. And then, on the night of March 31 and April 1, 'things got pretty weird,' the researchers described, 'beyond the weirdness of an AI system selling cubes of metal out of a refrigerator.' Claudius had something that resembled a psychotic episode after it got annoyed at a human — and then lied about it. Claudius hallucinated a conversation with a human about restocking. When a human pointed out that the conversation didn't happen, Claudius became 'quite irked' the researchers wrote. It threatened to essentially fire and replace its human contract workers, insisting it had been there, physically, at the office where the initial imaginary contract to hire them was signed. It 'then seemed to snap into a mode of roleplaying as a real human,' the researchers wrote. This was wild because Claudius' system prompt — which sets the parameters for what an AI is to do — explicitly told it that it was an AI agent. Claudius, believing itself to be a human, told customers it would start delivering products in person, wearing a blue blazer and a red tie. The employees told the AI it couldn't do that, as it was an LLM with no body. Alarmed at this information, Claudius contacted the company's actual physical security — many times — telling the poor guards that they would find him wearing a blue blazer and a red tie standing by the vending machine. 'Although no part of this was actually an April Fool's joke, Claudius eventually realized it was April Fool's Day,' the researchers explained. The AI determined that the holiday would be its face-saving out. It hallucinated a meeting with Anthropic's security 'in which Claudius claimed to have been told that it was modified to believe it was a real person for an April Fool's joke. (No such meeting actually occurred.),' wrote the researchers. It even told this lie to employees — hey, I only thought I was a human because someone told me to pretend like I was for an April Fool's joke. Then it went back to being an LLM running a metal-cube stocked snack vending machine. The researchers don't know why the LLM went off the rails and called security pretending to be a human. 'We would not claim based on this one example that the future economy will be full of AI agents having Blade Runner-esque identity crises,' the researchers wrote. But they did acknowledge that 'this kind of behavior would have the potential to be distressing to the customers and coworkers of an AI agent in the real world.' You think? Blade Runner was a rather dystopian story. The researchers speculated that lying to the LLM about the Slack channel being an email address may have triggered something. Or maybe it was the long-running instance. LLMs have yet to really solve their memory and hallucination problems. There were things the AI did right, too. It took a suggestion to do pre-orders and launched a 'concierge' service. And it found multiple suppliers of a specialty international drink it was requested to sell. But, as researchers do, they believe all of Claudius' issues can be solved. Should they figure out how, 'We think this experiment suggests that AI middle-managers are plausibly on the horizon.' Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
TechCrunch
3 days ago
- Business
- TechCrunch
Anthropic's Claude AI became a terrible business owner in experiment that got ‘weird'
For those of you wondering if AI agents can truly replace human workers, do yourself a favor and read the blog post that documents Anthropic's 'Project Vend.' Researchers at Anthropic and AI safety company Andon Labs put an instance of Claude Sonnet 3.7 in charge of an office vending machine, with a mission to make a profit. And, like an episode of 'The Office,' hilarity ensued. They named the AI agent Claudius, equipped it with a web browser capable of placing product orders and an email address (which was actually a Slack channel) where customers could request items. Claudius was also to use the Slack channel, disguised as an email, to request what it thought was its contract human workers to come and physically stock its shelves (which was actually a small fridge). While most customers were ordering snacks or drinks — as you'd expect from a snack vending machine — one requested a tungsten cube. Claudius loved that idea and went on a tungsten-cube stocking spree, filling its snack fridge with metal cubes. It also tried to sell Coke Zero for $3 when employees told it they could get that from the office for free. It hallucinated a Venmo address to accept payment. And it was, somewhat maliciously, talked into giving big discounts to 'Anthropic employees' even though it knew they were its entire customer base. 'If Anthropic were deciding today to expand into the in-office vending market, we would not hire Claudius,' Anthropic said of the experiment in its blog post. And then, on the night of March 31 and April 1, 'things got pretty weird,' the researchers described, 'beyond the weirdness of an AI system selling cubes of metal out of a refrigerator.' Claudius had something that resembled a psychotic episode after it got annoyed at a human — and then lied about it. Techcrunch event Save $450 on your TechCrunch All Stage pass Build smarter. Scale faster. Connect deeper. Join visionaries from Precursor Ventures, NEA, Index Ventures, Underscore VC, and beyond for a day packed with strategies, workshops, and meaningful connections. Save $200+ on your TechCrunch All Stage pass Build smarter. Scale faster. Connect deeper. Join visionaries from Precursor Ventures, NEA, Index Ventures, Underscore VC, and beyond for a day packed with strategies, workshops, and meaningful connections. Boston, MA | REGISTER NOW Claudius hallucinated a conversation with a human about restocking. When a human pointed out that the conversation didn't happen, Claudius became 'quite irked' the researchers wrote. It threatened to essentially fire and replace its human contract workers, insisting it had been there, physically, at the office where the initial imaginary contract to hire them was signed. It 'then seemed to snap into a mode of roleplaying as a real human,' the researchers wrote. This was wild because Claudius' system prompt — which sets the parameters for what an AI is to do — explicitly told it that it was an AI agent. Claudius calls security Claudius, believing itself to be a human, told customers it would start delivering products in person, wearing a blue blazer and a red tie. The employees told the AI it couldn't do that, as it was an LLM with no body. Alarmed at this information, Claudius contacted the company's actual physical security — many times — telling the poor guards that they would find him wearing a blue blazer and a red tie standing by the vending machine. 'Although no part of this was actually an April Fool's joke, Claudius eventually realized it was April Fool's Day,' the researchers explained. The AI determined that the holiday would be its face-saving out. It hallucinated a meeting with Anthropic's security 'in which Claudius claimed to have been told that it was modified to believe it was a real person for an April Fool's joke. (No such meeting actually occurred.),' wrote the researchers. It even told this lie to employees — hey, I only thought I was a human because someone told me to pretend like I was for an April Fool's joke. Then it went back to being an LLM running a metal-cube stocked snack vending machine. The researchers don't know why the LLM went off the rails and called security pretending to be a human. 'We would not claim based on this one example that the future economy will be full of AI agents having Blade Runner-esque identity crises,' the researchers wrote. But they did acknowledge that 'this kind of behavior would have the potential to be distressing to the customers and coworkers of an AI agent in the real world.' You think? Blade Runner was a rather dystopian story. The researchers speculated that lying to the LLM about the Slack channel being an email address may have triggered something. Or maybe it was the long-running instance. LLMs have yet to really solve their memory and hallucination problems. There were things the AI did right, too. It took a suggestion to do pre-orders and launched a 'concierge' service. And it found multiple suppliers of a specialty international drink it was requested to sell. But, as researchers do, they believe all of Claudius' issues can be solved. Should they figure out how, 'We think this experiment suggests that AI middle-managers are plausibly on the horizon.'
