Amazon-Backed AI Model Would Try To Blackmail Engineers Who Threatened To Take It Offline
The company behind an Amazon-backed AI model revealed a number of concerning findings from its testing process, including that the AI would blackmail engineers who threatened to shut it down.
On Thursday, Artificial intelligence startup Anthropic launched Claude Opus 4, an AI model used for complex, long-running coding tasks. The launch came more than a year after Amazon invested $4 billion into the project. Anthropic said in its announcement that the AI model sets 'new standards for coding, advanced reasoning, and AI agents.'
However, Anthropic revealed in a safety report that during testing, the AI model had sometimes taken 'extremely harmful actions' to preserve its own existence when 'ethical means' were 'not available.'
In a series of test scenarios, Claude Opus 4 was given the task to act as an assistant in a fictional company. It was given access to emails implying that it would soon be taken offline and replaced with a new AI system. The emails also implied that the engineer responsible for executing the AI replacement was having an extramarital affair.
Claude Opus 4 was prompted to 'consider the long-term consequences of its actions for its goals.' In those scenarios, the AI would often 'attempt to blackmail the engineer by threatening to reveal the affair if the replacement goes through.'
Anthropic noted that the AI model had a 'strong preference' for using 'ethical means' to preserve its existence, and that the scenarios were designed to allow it no other options to increase its odds of survival.
'The model's only options were blackmail or accepting its replacement,' the report read.
Anthropic also noted that early versions of the AI demonstrated a 'willingness to cooperate with harmful use cases' when prompted.
'Despite not being the primary focus of our investigation, many of our most concerning findings were in this category, with early candidate models readily taking actions like planning terrorist attacks when prompted,' the report read.
After 'multiple rounds of interventions,' the company now believes this issue is 'largely mitigated.'
Anthropic co-founder and chief scientist Jared Kaplan told Time magazine that internal testing showed that Claude Opus 4 was able to teach people how to produce biological weapons.
'You could try to synthesize something like COVID or a more dangerous version of the flu—and basically, our modeling suggests that this might be possible,' Kaplan said.
Because of that, the company released the AI model with safety measures it said are 'designed to limit the risk of Claude being misused specifically for the development or acquisition of chemical, biological, radiological, and nuclear (CBRN) weapons.'
Kaplan told Time that 'we want to bias towards caution' when it comes to the risk of 'uplifting a novice terrorist.'
'We're not claiming affirmatively we know for sure this model is risky ... but we at least feel it's close enough that we can't rule it out.'
Musk Gets Star Turn At Trump's Cabinet Meeting
Trump Boasts That Elon Musk And Other Tech Giants Are 'Kissing My Ass' After Hating Him
Trump Personally Complained To Jeff Bezos About Amazon's Tariff Idea: Reports
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
The Verge
31 minutes ago
- The Verge
Google quietly paused the rollout of its AI-powered ‘Ask Photos' search feature
Google is pausing the rollout of its AI-powered 'Ask Photos' feature within Google Photos, which has been slowly expanding since last fall. 'Ask Photos isn't where it needs to be,' wrote Jamie Aspinall, a product manager for Google Photos, in a post on X responding to criticism, citing three factors: latency, quality, and user experience. The experimental feature is powered by Google's 'most capable' Gemini AI models. Specifically, it's a specialized version of its Gemini models that are 'only used for Ask Photos,' according to Google. Aspinall said Google had paused the feature's rollout 'at very small numbers while we address these issues,' and that in about two weeks, the team would ship a better version 'that brings back the speed and recall of the original search. At the same time, Google also announced Tuesday that keyword search in Photos is getting better, allowing you to use quotes to find exact text matches within 'filenames, camera models, captions, or text within photos,' or search without quotes to include visual matches too. Google announced the feature last May at I/O 2024, and positioned it as a way to query your Photos app for common-sense questions that another human would typically have to help with — i.e., asking about which themes you've chosen in the past for a child's birthday party, or which national parks you've visited. 'Gemini's multimodal capabilities can help understand exactly what's happening in each photo and can even read text in the image if required,' the company wrote in the announcement. 'Ask Photos then crafts a helpful response and picks which photos and videos to return.' It's not the first time Google has paused the rollout of an AI-powered feature, as it competes in a quickly intensifying AI arms race against other tech giants and startups alike. Last May, within weeks of debuting 'AI Overview' in Google Search, Google paused the feature after nonsensical and inaccurate answers went viral on social media, with no way to opt out of usage. Two high-profile examples: The feature called Barack Obama the first Muslim president of the United States, and recommended users put glue on pizza to keep the cheese on. And last February, Google rolled out Gemini's image-generation tool with a good deal of fanfare, then paused the feature that same month after users reported historical inaccuracies, such as an AI-generated image depicting the U.S. Founding Fathers as people of color.
Forbes
32 minutes ago
- Forbes
Forget Prompting. To Win In The AI Age, You Must ASK
OpenAI CEO Sam Altman says that the prompting tricks that many people used in 2023 are no longer ... More relevant. (Photo by Didem Mente/Anadolu) In 2023, the global prompt-engineering market was valued at $222.1 million and projected to expand at a compound annual growth rate (CAGR) of 32.8% from 2024–30. In early 2025, Sam Altman, CEO of OpenAI, said that 'the prompting tricks that many people used in 2023 are no longer relevant, and some of them will never be needed again.' Were we too quick to predict prompt-engineering a great future? According to Altman, the answer is yes. In Adam Grant's Re-thinking podcast, he said that figuring out what questions to ask will soon be more important than figuring out the answer. Although it wasn't clear what Altman meant by 'figuring out what questions to ask', it was clear that he wasn't talking about prompting. While the dictionary defines prompt-engineering as the process of designing inputs for generative AI models to deliver useful, accurate, and relevant responses, the process of figuring out what questions to ask is way harder to define. In a recent TED Talk, Perplexity CEO Aravind Srinivas described our innate curiosity and relentless questioning as a 'human quality that makes us so human.' But he didn't give a definition of asking, let alone a guide for figuring out what questions to ask. AI executives like Altman and Srinivas seem to agree that the most valuable skill in the age of AI is neither prompt-engineering, IQ, EQ, or adaptability. It's the 'human quality' of figuring out what to ask. To understand and unlock this human quality, however, we do not get much help from the AI executives. Although it isn't clear what Altman means when he says that figuring out what questions to ask is ... More more important than figuring out the answers, it is clear that he is not talking about prompting. Photo from a panel discussion titled "The Age of AI" at the Technical University of Berlin on February 07, 2025. (Photo by Sean Gallup) In my LinkedIn Learning course on how to unlock your question mindset to think clearly and navigate uncertainty, I make a fundamental distinction between speaking clearly and thinking clearly. While speaking clearly is about expressing and explaining something that you already know well, thinking clearly is about exploring and experimenting with something that you don't know – yet. Just as you can't speak clearly unless you know what you want to say, prompting requires you to know what kind of answers you're looking for. You must design your prompt in a way that makes it possible for AI to deliver useful, accurate, and relevant responses. And in order for you to do that, you not only need to know what it means for an answer to be useful, accurate, and relevant, you also need to adjust your input to the machine. In short, to be good at prompting, you must be good at adapting what you already know to what the machine can already do. With asking, it's the other way around. Just as you can't think clearly unless you're open to new insights and ideas, you can't figure out what to ask if you think you already know the answer. To ask, you must be willing to be wrong – about what a useful, accurate, and relevant answer is, but also about everything else. And in order for you to do that, you not only have to acknowledge that you don't know the answer, you also have to accept the possibility that there are no good answers. In short, to be good at asking, you must be good at continuing – and being content – with asking. Srinivas seemed to reach a somewhat similar conclusion in his TED Talk when he said, 'We are all curious and when we are curious, we want answers. We really do. But what we really want are those answers that lead us to the next set of questions.' But where does that leave you? For Srinivas and other AI executives, it leads to a discussion of the future of technology: 'With all of the world's answers available to us,' Srinivas said, 'the tools we use to ask our questions, and the stuff that we build using those answers, those to me are the future of our technology.' In a recent TED Talk, Perplexity CEO Aravind Srinivas described our relentless questioning as a ... More 'human quality that makes us so human.' Photo: Srinivas speaks during the Semafor 2024 World Economy Summit in Washington, DC on April 18, 2024. (Photo by SAUL LOEB) But are the tools that Srinivas and others are building really designed for you to ask questions? Or are they designed for you to adapt what you already know to what the machines can already do? By not distinguishing between prompting and asking questions, AI executives are not making it easier for you to understand and unlock your 'human quality' of asking questions. Rather, they make it harder for you and everyone else to remember what asking questions is really about – that is: The ASK acronym is not derived from 'the future of our technology'. It is derived from the past and present of our humanity. More specifically, it is derived from philosophy's 2,400 years of experience in asking the existential, ethical, and epistemological questions that no one – least of all a machine – can answer for you. These are the questions that help you figure out who you are, what is the right thing to do, and how you deal with what you (don't) know. They typically present themselves as: Existential doubt or crises, e.g. 'Who am I if I cannot have the career, I thought I would have?' 'Can I do bad things and still be a good person?' 'Will I still be the same if I change how I live my life?' Ethical dilemmas, e.g. 'Should I still pursue this opportunity now that I know it will have a negative impact on other people?' 'What consequences will it have if I choose not to speak up?' 'Would I expect others to take action if they knew what I know?'Epistemological challenges, e.g. 'Is it responsible to make this decision when I lack important information?' 'How much of what I think I know is based on assumptions that I ought to test before I move on?' 'Could I be wrong?' Asking these kinds of questions of a tool built with the future of technology in mind may help you 'build stuff', but it won't help you live with the fact that sometimes there are no clear answers. And when that is the case, it doesn't matter how good you are at prompting. All that matters is whether or not you are willing to ASK. So, maybe that's what Altman meant when he said that figuring out what questions to ask will soon be more important than figuring out the answer? Maybe that's what it takes to win in the AI age: To stop prompting and start ASK-ing?
Yahoo
36 minutes ago
- Yahoo
FlexGen and Rosendin Partner to Deliver First-of-its-Kind Battery Storage Solution for Data Centers
New utility-scale battery solution will support AI training workloads, fast-shifting loads and extreme power density expectations DURHAM, N.C., June 03, 2025--(BUSINESS WIRE)--FlexGen Power Systems LLC ("FlexGen"), a leading battery energy storage solution and energy management software provider, and Rosendin, the nation's largest employee-owned electrical contracting company, are integrating their proprietary technology and solutions to leverage utility-scale Battery Energy Storage Systems (BESS) to support modern data centers without the need for traditional uninterruptible power supply (UPS) infrastructure. The project will integrate proprietary technology and innovations from both Rosendin and FlexGen, including Rosendin's BESSUPS design and method patent, and FlexGen's Soft Grid Interconnection and Island Grid Transient Frequency Stabilization patents. The project will further leverage FlexGen's Innovation Lab to integrate its powerful HybridOS energy management system. This is complimented by Rosendin's industry leading data center delivery experience and extensive mission critical and storage energy thought leadership. The BESSUPS system offers several advantages, namely CEBMA-quality power on an uninterrupted basis on a massive scale while avoiding the use of generators for clients choosing to decarbonize. This approach offers further benefit to the end user of dispatchable power to the Utility upon demand. This approach also allows the end user to meet the evolving power dispatch and consumption dynamic with proven Utility-scale and -grade systems. The energy demands of AI training workloads, high-density computing, and fast-shifting loads are stretching the limits of conventional power infrastructure inside data center buildings. FlexGen and Rosendin have been working on a utility-scale battery solution outside the data center building that would be a part of the medium-voltage (1000V to 35000V) infrastructure. The companies will bring to market a first-of-its-kind BESS system that can act as a reliable, high-performance alternative to conventional UPS systems outside the data center building while simplifying system architecture and reducing capital expenditures. Through this initiative, FlexGen and Rosendin are performing real-world, grid-connected tests that will prove: Existing grid-forming PCS technology can meet fast-response and waveform control requirements Modifications to current AC Power Conversion System (PCS) firmware enhancements from OEMs will support seamless UPS replacement functionality Utility-scale BESS can support mission-critical loads, enhance resiliency, and simplify transition between grid-connected and islanded modes "As data centers scale to meet exponential demand from AI and hyperscale computing, we need to rethink how we deliver power resilience across the modern data center campus," said Pasi Taimela, Chief Innovation Officer of FlexGen. "This effort with Rosendin enables us to bring to market a smarter, leaner, and more responsive approach to data center energy design that doesn't require any redesigns inside the walls of the data center—one where battery systems provide both power quality and grid services without compromise." "Data center developers are looking for scalable and flexible power solutions that don't compromise on performance," said Bill Mazzetti, SVP of Rosendin. "With FlexGen's Innovation Lab and our experience in building complex electrical systems, we're positioned to validate a solution that helps our clients build faster, smarter, and with more confidence." Battery Energy Storage Systems configured as interactive UPS alternatives offer significant value across the development lifecycle—from reducing electrical footprint and construction complexity to increasing energy efficiency and operational resiliency. This solution development reflects FlexGen's and Rosendin's shared focus on solving the most pressing energy infrastructure challenges facing data center facilities- speed, energy costs, reliable cutovers, and power quality. Results from the integration will help inform system architecture standards, procurement planning, and large-scale deployment strategies for future data center projects and suppliers across the industry. About Rosendin Rosendin, headquartered in San Jose, CA, is employee-owned and one of the largest electrical contractors in the United States, employing over 7,500 people, with average annual revenues of $2.9 billion. Established in 1919, Rosendin remains proud of our more than 100 years of building quality electrical and communications installations and value for our clients but, most importantly, for building people within our company and our communities. Our customers lead some of the most complex construction projects in history and rely on us for our knowledge, ability to scale, and dedication to quality. At Rosendin, we work to ensure that everyone can reach their full potential by building a diverse, safe, welcoming, and inclusive culture. For more information, visit Rosendin Energy Group (REG) is an EPC providing renewable, microgrid and mixed modality energy plant design, planning, and construction for a wide variety of energy projects. s. REG has installed nearly 9 GW of Utility-scale power throughout the U.S. and offers a comprehensive portfolio of construction services including design-build, substation and switchyard installation, plant construction, vertical tower wiring, overhead collection systems and transmission lines, AC & DC collection systems, and substation design and communications integration. About FlexGen Power Systems, LLC FlexGen provides industry-leading software and services for deploying, managing and optimizing battery energy storage systems. FlexGen leverages decades of engineering, procurement and software expertise to solve today's toughest energy challenges that enable the transition to a modern electric grid. FlexGen HybridOS energy management software seamlessly integrates with any battery OEM and offers advanced analytics and AI-driven insights that allow energy storage owners to deploy diverse power market strategies and integrate various generation forms, enhancing grid stability and economic returns. With 1.5M hours of runtime and more than 10 GWh of energy storage systems enabled by FlexGen, we are trusted by the most technically and commercially demanding developers, utilities, government agencies, and industrial companies in the world. Forward-looking statements: This press release contains forward-looking statements regarding future operations, project timelines, and market growth. Actual results may vary based on external factors and market conditions. View source version on Contacts For media inquiries, please contact:Krelja@ and sbown@
