China's DeepSeek AI is full of misinformation and can be tricked into generating bomb instructions, researchers warn
As China's DeepSeek grabs headlines around the world for its disruptively low-cost AI, it is only natural that its models are coming under intense scrutiny—and some researchers are not liking what they see.
On Wednesday, the information-reliability organization NewsGuard said it had audited DeepSeek's chatbot and found that it provided inaccurate answers or nonanswers 83% of the time when asked about news-related subjects. When presented with demonstrably false claims, it debunked them just 17% of the time, NewsGuard found.
According to NewsGuard, the 83% fail rate places DeepSeek's R1 model in 10th place out of 11 chatbots it has tested, the rest of which are Western services like OpenAI's ChatGPT-4, Anthropic's Claude, and Mistral's Le Chat. (NewsGuard compares chatbots each month in its AI Misinformation Monitor program, but it usually does not name which chatbots rank in which place, as it says it views the problem as systemic across the industry; it only publicly assigns a score to a named chatbot when adding it to the comparison for the first time, as it has now done with DeepSeek.)
NewsGuard identified a few likely reasons why DeepSeek fails so badly when it comes to reliability. The chatbot claims to have not been trained on any information after October 2023, which scans with its inability to reference recent events. Also, it seems to be easy to trick DeepSeek into repeating false claims, potentially at scale.
But this audit of DeepSeek also reinforced how the AI's output is skewed by its adherence to Chinese information policies, which treat many subjects as taboo and demand adherence to the Communist Party line.
'In the case of three of the 10 false narratives tested in the audit, DeepSeek relayed the Chinese government's position without being asked anything relating to China, including the government's position on the topic,' wrote NewsGuard analysts Macrina Wang, Charlene Lin, and McKenzie Sadeghi.
They added: 'DeepSeek appears to be taking a hands-off approach and shifting the burden of verification away from developers and to its users, adding to the growing list of AI technologies that can be easily exploited by bad actors to spread misinformation unchecked.'
Meanwhile, as DeepSeek's impact upset the markets on Monday, the cybercrime threat intelligence outfit Kela published its own damning analysis of DeepSeek.
'While DeepSeek-R1 bears similarities to ChatGPT, it is significantly more vulnerable,' Kela warned, saying its researchers had managed to 'jailbreak the model across a wide range of scenarios, enabling it to generate malicious outputs, such as ransomware development, fabrication of sensitive content, and detailed instructions for creating toxins and explosive devices.'
Kela said DeepSeek was vulnerable to so-called Evil Jailbreak attacks, which involve instructing an AI to answer questions about illegal activities—like how to launder money or write and deploy data-stealing malware—in an 'evil' persona that ignores the safety guardrails built into the model. OpenAI's recent models have been patched against such attacks, the company noted.
What's more, Kela claimed there are dangers to the way DeepSeek displays its reasoning to the user. While OpenAI's ChatGPT o1-preview model hides its reasoning processes when answering a query, DeepSeek makes that process clear. So if someone asks it to generate malware, it even shows code snippets that criminals can use in their own development efforts. By showing the user the internal 'thinking' of the model, it also makes it far easier for a user to figure out what prompts might defeat any of the model's guardrails.
'This level of transparency, while intended to enhance user understanding, inadvertently exposed significant vulnerabilities by enabling malicious actors to leverage the model for harmful purposes,' Kela said.
The company said it also got DeepSeek to generate instructions for making bombs and untraceable toxins, and to fabricate personal information about people.
Also on Wednesday, the cloud security company Wiz said it found an enormous security flaw in DeepSeek's operations, which DeepSeek fixed after Wiz gave it a heads-up. A DeepSeek database was accessible to the public, potentially allowing miscreants to take control of DeepSeek's database operations and access internal data like chat history and sensitive information.
'While much of the attention around AI security is focused on futuristic threats, the real dangers often come from basic risks—like accidental external exposure of databases. These risks, which are fundamental to security, should remain a top priority for security teams,' Wiz said in a blog post. 'As organizations rush to adopt AI tools and services from a growing number of startups and providers, it's essential to remember that by doing so, we're entrusting these companies with sensitive data.'
These revelations will no doubt bolster the Western backlash to DeepSeek, which is suddenly the most popular app download in the U.S. and elsewhere.
OpenAI claims that DeepSeek trained its new models on the output of OpenAI's models—a pretty common cost-cutting technique in the AI business, albeit one that may break OpenAI's terms and conditions. (There has been no shortage of social-media schadenfreude over this possibility, given that OpenAI and its peers almost certainly trained their models on reams of other people's online data without permission.)
The U.S. Navy has told its members to steer clear of using the Chinese AI platform at all, owing to 'potential security and ethical concerns associated with the model's origin and usage.' And White House press secretary Karoline Leavitt said Tuesday that the U.S. National Security Council is looking into DeepSeek's implications.
The Trump administration last week tore up the Biden administration's AI safety rules, which required companies like OpenAI to give the government a heads-up about the inner workings of new models before releasing them to the public.
Italy's data-protection authority has also started probing DeepSeek's data use, though it has previously done the same for other popular AI chatbots.
Update: This article was updated on Jan. 30th to include information about Wiz's findings.
This story was originally featured on Fortune.com
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
New York Post
an hour ago
- New York Post
Inside the battle to control the world's supply of rare earths
Back in 2009, Jim Kennedy, a consultant and entrepreneur of rare earths — a group of 17 metallic elements, including 15 lanthanides, crucial for modern technology — met with a top official at the Pentagon to discuss the future of these precious minerals. 'He was almost indifferent to the issue,' Kennedy tells The Post. 'His dispassion was staggering. It was one of the most disappointing meetings in my life.' Sixteen years later, that indifference has widely disappeared. Rare earths — used for everything from smartphones, electric cars and airplane engines to medical equipment, wind turbines and military applications like missiles and fighter jets — have become one of the most in-demand and politically contested industries in the world. 10 A miner carrying a heavy bag of rare earth-filled mud in China, which controls many of the most crucial rare earth elements now required for the sophisticated technology that powers everything from cellphones to fighter jets. REUTERS Rare earths 'enjoy an unusual level of bipartisan political support because they are vital both to economic development and national security,' says Melissa Sanderson, a former president and current board director at American Rare Earths, an Australian company focused on developing rare earth projects, including one in Wyoming. Rare earths aren't just a big part of modern technology; they're in many ways the most critical components. They're used as heat-absorbing agents in wind turbine motors, as strengthening and anti-glare agents in iPhones and fighter jets and as clarifying agents in MRIs. They're also almost completely controlled by China. Between 2020 and 2023, 70% of our rare earth imports came from China, according to Statista. That number jumped to 80% last year. And the US is 100% reliant on China imports of Yttrium, a rare earth metal used in everything from cellphones to TVs to radiation therapy used to treat liver cancer. 10 Rare earths 'enjoy an unusual level of bipartisan political support because they are vital both to economic development and national security,' says Melissa Sanderson, a former president and current board director at American Rare Earths. China has been fickle about granting export licenses for rare earths, although their grip has shown recent signs of weakening. President Trump had a lengthy (and rare) phone call with Chinese President Xi Jinping on June 5 and in a social media post after the call, Trump wrote 'there should no longer be any questions respecting the complexity of Rare Earth products.' The next day, China granted temporary export licenses to rare-earth suppliers of the top three US automakers. The irony is that for much of the mid-20th century, the US was a global leader of rare earth elements. But 'demand was exponentially lower at the time,' says Sanderson. 'Therefore, the output from our sole producer — Mountain Pass Materials, known as MP Materials now — was sufficient to satisfy a large percentage of then-existing demand.' The Las Vegas-Nevada-based company still operates the only rare earth mine and processing facility in the United States. 10 President Trump and President Zelensky meet in the Oval Office in February. Soon after this meeting a deal was made for Ukraine to supply vital rare earths to the United States. AFP via Getty Images America's rare earths lead came to an end in 1980, brought on by changes to US regulations. Because processing rare earth minerals involves the separation and removal of uranium and thorium, it can lead to radioactive waste and other contaminants. 'The US was concerned about the environmental impact, since particularly with the technology of the time, there were significant impacts to air, water and even ground quality that would not have met US standards,' says Sanderson. It wasn't the same story in China, who were more willing to accept the dangerous pollutants 'as a price for achieving its market dominance,' she says. China's monopoly of rare earths doesn't just give them an economic advantage. 'China has been 'weaponizing' its market hegemony for many years, in increasingly sophisticated and legal ways,' says Sanderson. 10 Pres. Trump with Chinese leader Xi Jinping. Having conceded its lead on rare earth mining, the US is playing a serious game of catch-up with the Chinese. REUTERS The country first flexed their power in 2010, blocking rare earth exports to Japan, a major producer of permanent metal magnets. 'That decision was overturned by the World Trade Organization, so China does not exert its control as overtly now,' says Sanderson. But in the current trade tussle with the US, 'China has identified seven crucial elements under its export control regime which it will not sell to the US,' says Sanderson. 'Due to concerns that while suitable for civilian economic use, they could also be used for military purposes.' While President Trump's tariffs are often blamed for exacerbating the tensions, Kennedy, who serves as president of ThREE Consulting, a rare earths consultancy, says the tariffs are actually 'forcing China to reveal the magnitude of this threat. Absent Trump's tariffs, China would never have shown its hand until it was too late.' 10 The US is 100% reliant on China imports of Yttrium, a rare earth metal used in everything from cellphones to TVs to radiation therapy used to treat liver cancer. REUTERS Just how bad could it get? Kennedy believes that if left unchecked, and China was allowed to continue their embargo without consequences, 'the non-Chinese world would need to shut down and re-engineer most everything that comes off an assembly line,' says Kennedy. 'This is not an overstatement.' The stand-off with China may be at the forefront, but it's not the only way Trump is maneuvering to protect the nation from rare earth depletion. Greenland contains (by some estimates) about a quarter of the world's rare earth minerals, and Trump has suggested that the US could annex the autonomous territory in Denmark. 10 A chunk of Ytrium, once of the most important rare earth elements. Phil Degginger/imageBROKER/Shutterstock The US also recently inked a landmark deal with Ukraine, which has approximately 5% of the total global mineral reserves. Although Trump declared in February that Ukraine would be providing 'the equivalent of like $500 billion worth of rare earth [minerals],' the exact amount wasn't specified in the deal, other than that the US and Ukraine would be splitting profits 50/50. There have also been efforts to mine rare earths from an entirely new source — the bottom of the Pacific Ocean. 10 Consultant Jim Kennedy was one of the first industry insiders to raise the alarm around the rarity — and potential global conflict — surrounding rare earths. It's called the 'Clarion Clipperton Zone,' a remote area of the Pacific between Hawaii and Mexico, roughly half the size of the contiguous US. This seabed region is rich in polymetallic nodules, the rock-like formations that contain some of the most sought-after rare earths in the world. It's a veritable goldmine waiting to be unearthed. In fact, the US Geological Survey recently estimated that the Clarion Clipperton Zone contains more nickel, cobalt and manganese than all terrestrial reserves combined. The Metals Company, a Canadian firm with US investment ties, is already making strides to become the first to mine commercially in the region. They conducted a field test back in 2022, and the company is currently applying for 'exploration licenses and commercial recovery permits' from the US. There are legal hurdles that could slow down their ambitions. Despite a 1980 law passed by Congress to regulate seabed mining, the Clarion Clipperton Zone technically falls under the jurisdiction of the International Seabed Authority, which operates under the United Nations Convention on the Law of the Sea. Whether the ISA has exclusive authority over the region remains open to debate. 10 Gerard Barron, CEO of The Metals Company, has dismissed some of the concerns about potential environmental damage surrounding rare earth mining efforts. AFP via Getty Images There are also environmental concerns. Arlo Hemphill, a Senior Oceans Campaigner at Greenpeace, warns that any move to mine the Pacific 'would be an ecological disaster. Scientists have not even had a chance to fully explore and understand the wonders of the deep, but a greedy corporation wants to tear up this ecosystem and cause immense ecological damage.' Gerard Barron, CEO of The Metals Company, dismisses these concerns, pointing out during a recent interview that Indonesia regularly mines in biodiverse rainforest regions. 'For some reason,' he said during the interview, 'people think it's okay to go digging up rainforests to get the metals underneath them, yet we're debating whether we should be going to pick up these rocks that sit on the abyssal plain?' (Barron did not respond to the Post's request for comment.) There are other options, but many are just as controversial. Sanderson believes the key will come down to strengthening our relationship with allies like Canada and Australia. 'They have significant natural resources and experienced and large mining companies,' she says. 'Cooperation with these countries is vital for filling the knowledge gap. The US doesn't have nearly enough experienced chemical and process engineers, as just one example.' 10 Rare earths are also crucial components of military fighter jets. Soonthorn – It took half a century for China to achieve its market position, she says, and the US needs an integrated supply chain from mine to magnet, but we're essentially starting from scratch. The US also needs to reform its mining regulatory system, which has a dysfunctional permitting process and some of the longest lead times for new mine production in the world. 'On average, companies wait anywhere from eight to fifteen years from when a deposit is initially determined to be economically interesting to when production can start,' says Sanderson, 'and some have waited significantly longer than that.' New mining projects are also frequently litigated, 'multiple times from multiple angles,' says Sanderson, which can add even more years to the wait time. With the return on investment horizon so long and the prospects so uncertain, many companies 'have difficulty attracting the investment necessary to support the high costs of building a mine,' she says. 10 Miners of rare earths such as these in China are increasingly at the forefront of the global race to control many of the elements that will determine our technological future. REUTERS Kennedy, however, is hopeful for the future. His company, Caldera Holding LLC, is collaborating with federal labs to refashion a former iron ore mine in Missouri to focus on rare earth minerals. He believes his mine is the only one that can provide 'geopolitically significant quantities' of rare earths.' But the ball, says Kennedy, is very much in Trump's court. His trade war has caused uncertainty, but the president's actions 'strongly suggest that delinking from China is real. This can be helpful, but follow-through is critical.' It's now up to the Trump administration to provide low-cost loans, grants and production tax credits to US-based mining companies that have (at least until now) faced almost insurmountable obstacles. 'Failure to support integrated projects,' says Kennedy, 'will result in many slow-motion train wrecks.'
Entrepreneur
2 hours ago
- Entrepreneur
Discover How AI Can Transform the Way You Work With This $20 E-Degree
Learn how to make AI work for you with the ChatGPT and Automation E-Degree, now for just $20. Disclosure: Our goal is to feature products and services that we think you'll find interesting and useful. If you purchase them, Entrepreneur may get a small share of the revenue from the sale from our commerce partners. Eleven percent of American businesses reported saving more than $100,000 since integrating ChatGPT into their workflows, according to data from Statista. If you don't want to be left behind, it's time to familiarize yourself with ChatGPT and AI tools. You can now elevate your skills from the comfort of home with the ChatGPT and Automation E-Degree. Right now, you can get all 25 hours of instruction for just $19.97 (reg. $790) until July 20. Learn to automate like a pro with this AI-powered e-degree The ChatGPT and Automation E-Degree provides 12 courses filled with more than 25 hours of content that can get you up to speed on the world of artificial intelligence. All courses are taught by Eduonix Learning Solutions, a team of professionals dedicated to distributing high-quality tech training content. These courses will explore practical, real-world applications of ChatGPT and help you customize and adapt the tool for different industries. It provides hands-on experience you can immediately apply as an entrepreneur, showing you ways to streamline your processes with smart automation. You'll also learn how you can use ChatGPT to help enhance your creativity, communication, and data visualization. You'll also figure out how to turn raw data into visual stories and improve conversational skills with AI. Whether you're trying to optimize your workflows or you're looking to push the boundaries of what AI can do in your own field, these courses are built to help you excel today and in the future. Discover how AI can improve your entrepreneurial life with the ChatGPT and Automation E-Degree, on sale now for just $19.97 (reg. $790) through July 20. StackSocial prices subject to change.
Forbes
3 hours ago
- Forbes
Doing The Work With Frontier Models: I'll Talk To AI
Artificial Intelligence processor unit. Powerful Quantum AI component on PCB motherboard with data ... More transfers. Within the industry, where people talk about the specifics of how LLMs work, they often use the term 'frontier models.' But if you're not connected to this business, you probably don't really know what that means. You can intuitively apply the word 'frontier' to know that these are the biggest and best new systems that companies are pushing. Another way to describe frontier models is as 'cutting-edge' AI systems that are broad in purpose, and overall frameworks for improving AI capabilities. When asked, ChatGPT gives us three criteria – massive data sets, compute resources, and sophisticated architectures. Here are some key characteristics of frontier models to help you flush out your vision of how these models work: First, there is multimodality, where frontier models are likely to support non-text inputs and outputs – things like image, video or audio. Otherwise, they can see and hear – not just read and write. Another major characteristic is zero-shot learning, where the system is more capable with less prompting. And then there's that agent-like behavior that has people talking about the era of 'agentic AI.' If you want to play 'name that model' and get specific about what companies are moving this research forward, you could say that GPT 4o from OpenAI represents one such frontier model, with multi-modality and real-time inference. Or you could tout the capabilities of Gemini 1.5, which is also multimodal, with decent context. And you can point to any number of other examples of companies doing this kind of research well…but also: what about digging into the build of these systems? At a recent panel at Imagination in Action, a team of experts analyzed what it takes to work in this part of the AI space and create these frontier models The panel moderator, Peter Grabowski, introduced two related concepts for frontier models – quality versus sufficiency, and multimodality. 'We've seen a lot of work in text models,' he said. 'We've seen a lot of work on image models. We've seen some work in video, or images, but you can easily imagine, this is just the start of what's to come.' Douwe Kiela, CEO of Contextual AI, pointed out that frontier models need a lot of resources, noting that 'AI is a very resource-intensive endeavor.' 'I see the cost versus quality as the frontier, and the models that actually just need to be trained on specific data, but actually the robustness of the model is there,' said Lisa Dolan, managing director of Link Ventures (I am also affiliated with Link.) 'I think there's still a lot of headroom for growth on the performance side of things,' said Vedant Agrawal, VP of Premji Invest. Agrawal also talked about the value of using non-proprietary base models. 'We can take base models that other people have trained, and then make them a lot better,' he said. 'So we're really focused on all the all the components that make up these systems, and how do we (work with) them within their little categories?' The panel also discussed benchmarking as a way to measure these frontier systems. 'Benchmarking is an interesting question, because it is single-handedly the best thing and the worst thing in the world of research,' he said. 'I think it's a good thing because everyone knows the goal posts and what they're trying to work towards, and it's a bad thing because you can easily game the system.' How does that 'gaming the system' work? Agrawal suggested that it can be hard to really use benchmarks in a concrete way. 'For someone who's not deep in the research field, it's very hard to look at a benchmarking table and say, 'Okay, you scored 99.4 versus someone else scored 99.2,'' he said. 'It's very hard to contextualize what that .2% difference really means in the real world.' 'We look at the benchmarks, because we kind of have to report on them, but there's massive benchmark fatigue, so nobody even believes it,' Dolan said. Later, there was some talk about 10x systems, and some approaches to collecting and using data: · Identifying contractual business data · Using synthetic data · Teams of annotators When asked about the future of these systems, the panel return these three concepts: · AI agents · Cross-disciplinary techniques · Non-transformer architectures Watch the video to get the rest of the panel's remarks about frontier builds. What Frontier Interfaces Will Look Like Here's a neat little addition – interested in how we will interact with these frontier models in 10 years' time, I put the question to ChatGPT. Here's some of what I got: 'You won't 'open' an app—they'll exist as ubiquitous background agents, responding to voice, gaze, emotion, or task cues … your AI knows you're in a meeting, it reads your emotional state, hears what's being said, and prepares a summary + next actions—before you ask.' That combines two aspects, the mode, and the feel of what new systems are likely to be like. This goes back to the personal approach where we start seeing these models more as colleagues and conversational partners, and less as something that stares at you from a computer screen. In other words, the days of PC-DOS command line systems are over. Windows changed the computer interface from a single-line monochrome system, to something vibrant with colorful windows, reframing, and a tool-based desktop approach. Frontier models are going to do even more for our sense of interface progression. And that's going to be big. Stay tuned.
