Latest news with #Mantzarlis
Yahoo
3 days ago
- Yahoo
Meta sues AI ‘nudify' app Crush AI for advertising on its platforms
Meta has sued the maker of a popular AI 'nudify' app, Crush AI, that reportedly ran thousands of ads across Meta's platforms. In addition to the lawsuit, Meta says it's taking new measures to crack down on other apps like Crush AI. In a lawsuit filed in Hong Kong, Meta alleged Joy Timeline HK, the entity behind Crush AI, attempted to circumvent the company's review process to distribute ads for AI nudify services. Meta said in a blog post that it repeatedly removed ads by the entity for violating its policies, but claims Joy Timeline HK continued to place additional ads anyway. Crush AI, which uses generative AI to make fake, sexually explicit images of real people without their consent, reportedly ran more than 8,000 ads for its 'AI undresser' services on Meta's platform in the first two weeks of 2025, according to the author of the Faked Up newsletter, Alexios Mantzarlis. In a January report, Mantzarlis claimed that Crush AI's websites received roughly 90% of their traffic from either Facebook or Instagram, and that he flagged several of these websites to Meta. Crush AI reportedly evaded Meta's ad review processes by setting up dozens of advertiser accounts and frequently changed domain names. Many of Crush AI's advertiser accounts, according to Mantzarlis, were named 'Eraser Annyone's Clothes' followed by different numbers. At one point, Crush AI even had a Facebook page promoting its service. Facebook and Instagram are hardly the only platforms dealing with such challenges. As social media companies like X and Meta race to add generative AI to their apps, they've also struggled to moderate how AI tools can make their platforms unsafe for users, particularly minors. Researchers have found that links to AI undressing apps soared in 2024 on platforms like X and Reddit, and on YouTube, millions of people were reportedly served ads for such apps. In response to this growing problem, Meta and TikTok have banned keyword searches for AI nudify apps, but getting these services off their platforms entirely has proven challenging. In a blog post, Meta said it has developed new technology to specifically identify ads for AI nudify or undressing services 'even when the ads themselves don't include nudity.' The company said it is now using matching technology to help find and remove copycat ads more quickly, and has expanded the list of terms, phrases and emoji that are flagged by its systems. Meta said it is also applying the tactics it has traditionally used to disrupt networks of bad actors to these new networks of accounts running ads for AI nudify services. Since the start of 2025, Meta said, it has disrupted four separate networks promoting these services. Outside of its apps, the company said it will begin sharing information about AI nudify apps through Tech Coalition's Lantern program, a collective effort between Google, Meta, Snap and other companies to prevent child sexual exploitation online. Meta says it has provided more than 3,800 unique URLs with this network since March. On the legislative front, Meta said it would 'continue to support legislation that empowers parents to oversee and approve their teens' app downloads.' The company previously supported the US Take It Down Act, and said it's now working with lawmakers to implement it. Sign in to access your portfolio
TechCrunch
4 days ago
- TechCrunch
Meta sues AI ‘nudify' app Crush AI for advertising on its platforms
Meta has sued the maker of a popular AI 'nudify' app, Crush AI, that reportedly ran thousands of ads across Meta's platforms. In addition to the lawsuit, Meta says it's taking new measures to crack down on other apps like Crush AI. In a lawsuit filed in Hong Kong, Meta alleged Joy Timeline HK, the entity behind Crush AI, attempted to circumvent the company's review process to distribute ads for AI nudify services. Meta said in a blog post that it repeatedly removed ads by the entity for violating its policies, but claims Joy Timeline HK continued to place additional ads anyway. Crush AI, which uses generative AI to make fake, sexually explicit images of real people without their consent, reportedly ran more than 8,000 ads for its 'AI undresser' services on Meta's platform in the first two weeks of 2025, according to the author of the Faked Up newsletter, Alexios Mantzarlis. In a January report, Mantzarlis claimed that Crush AI's websites received roughly 90% of their traffic from either Facebook or Instagram, and that he flagged several of these websites to Meta. Crush AI reportedly evaded Meta's ad review processes by setting up dozens of advertiser accounts and frequently changed domain names. Many of Crush AI's advertiser accounts, according to Mantzarlis, were named 'Eraser Annyone's Clothes' followed by different numbers. At one point, Crush AI even had a Facebook page promoting its service. Facebook and Instagram are hardly the only platforms dealing with such challenges. As social media companies like X and Meta race to add generative AI to their apps, they've also struggled to moderate how AI tools can make their platforms unsafe for users, particularly minors. Researchers have found that links to AI undressing apps soared in 2024 on platforms like X and Reddit, and on YouTube, millions of people were reportedly served ads for such apps. In response to this growing problem, Meta and TikTok have banned keyword searches for AI nudify apps, but getting these services off their platforms entirely has proven challenging. In a blog post, Meta said it has developed new technology to specifically identify ads for AI nudify or undressing services 'even when the ads themselves don't include nudity.' The company said it is now using matching technology to help find and remove copycat ads more quickly, and has expanded the list of terms, phrases and emoji that are flagged by its systems. Techcrunch event Save $200+ on your TechCrunch All Stage pass Build smarter. Scale faster. Connect deeper. Join visionaries from Precursor Ventures, NEA, Index Ventures, Underscore VC, and beyond for a day packed with strategies, workshops, and meaningful connections. Save $200+ on your TechCrunch All Stage pass Build smarter. Scale faster. Connect deeper. Join visionaries from Precursor Ventures, NEA, Index Ventures, Underscore VC, and beyond for a day packed with strategies, workshops, and meaningful connections. Boston, MA | REGISTER NOW Meta said it is also applying the tactics it has traditionally used to disrupt networks of bad actors to these new networks of accounts running ads for AI nudify services. Since the start of 2025, Meta said, it has disrupted four separate networks promoting these services. Outside of its apps, the company said it will begin sharing information about AI nudify apps through Tech Coalition's Lantern program, a collective effort between Google, Meta, Snap and other companies to prevent child sexual exploitation online. Meta says it has provided more than 3,800 unique URLs with this network since March. On the legislative front, Meta said it would 'continue to support legislation that empowers parents to oversee and approve their teens' app downloads.' The company previously supported the US Take It Down Act, and said it's now working with lawmakers to implement it.
France 24
12-05-2025
- Health
- France 24
'Tool for grifters': AI deepfakes push bogus sexual cures
The rise of generative AI has made it easy -- and financially lucrative -- to mass-produce such videos with minimal human oversight, often featuring fake celebrity endorsements of bogus and potentially harmful products. In some TikTok videos, carrots are used as a euphemism for male genitalia, apparently to evade content moderation policing sexually explicit language. "You would notice that your carrot has grown up," the muscled man says in a robotic voice in one video, directing users to an online purchase link. "This product will change your life," the man adds, claiming without evidence that the herbs used as ingredients boost testosterone and send energy levels "through the roof." The video appears to be AI-generated, according to a deepfake detection service recently launched by the Bay Area-headquartered firm Resemble AI, which shared its results with AFP. "As seen in this example, misleading AI-generated content is being used to market supplements with exaggerated or unverified claims, potentially putting consumers' health at risk," Zohaib Ahmed, Resemble AI's chief executive and co-founder, told AFP. "We're seeing AI-generated content weaponized to spread false information." 'Cheap way' The trend underscores how rapid advances in artificial intelligence have fueled what researchers call an AI dystopia, a deception-filled online universe designed to manipulate unsuspecting users into buying dubious products. They include everything from unverified -- and in some cases, potentially harmful -- dietary supplements to weight loss products and sexual remedies. "AI is a useful tool for grifters looking to create large volumes of content slop for a low cost," misinformation researcher Abbie Richards told AFP. "It's a cheap way to produce advertisements," she added. Alexios Mantzarlis, director of the Security, Trust, and Safety Initiative at Cornell Tech, has observed a surge of "AI doctor" avatars and audio tracks on TikTok that promote questionable sexual remedies. Some of these videos, many with millions of views, peddle testosterone-boosting concoctions made from ingredients such as lemon, ginger and garlic. More troublingly, rapidly evolving AI tools have enabled the creation of deepfakes impersonating celebrities such as actress Amanda Seyfried and actor Robert De Niro. "Your husband can't get it up?" Anthony Fauci, former director of the National Institute of Allergy and Infectious Diseases, appears to ask in a TikTok video promoting a prostate supplement. But the clip is a deepfake, using Fauci's likeness. 'Pernicious' Many manipulated videos are created from existing ones, modified with AI-generated voices and lip-synced to match what the altered voice says. "The impersonation videos are particularly pernicious as they further degrade our ability to discern authentic accounts online," Mantzarlis said. Last year, Mantzarlis discovered hundreds of ads on YouTube featuring deepfakes of celebrities -- including Arnold Schwarzenegger, Sylvester Stallone, and Mike Tyson -- promoting supplements branded as erectile dysfunction cures. The rapid pace of generating short-form AI videos means that even when tech platforms remove questionable content, near-identical versions quickly reappear -- turning moderation into a game of whack-a-mole. Researchers say this creates unique challenges for policing AI-generated content, requiring novel solutions and more sophisticated detection tools. AFP's fact checkers have repeatedly debunked scam ads on Facebook promoting treatments -- including erectile dysfunction cures -- that use fake endorsements by Ben Carson, a neurosurgeon and former US cabinet member. Yet many users still consider the endorsements legitimate, illustrating the appeal of deepfakes. "Scammy affiliate marketing schemes and questionable sex supplements have existed for as long as the internet and before," Mantzarlis said. "As with every other bad thing online, generative AI has made this abuse vector cheaper and quicker to deploy at scale."
Yahoo
27-03-2025
- Politics
- Yahoo
How a TikTok network spreading Spanish-language immigration misinformation gained traction
A rumor falsely claiming green card holders are banned from leaving and re-entering the U.S. spread like wildfire on TikTok this week, garnering more than 21 million views across dozens of videos shared by a network of nearly 40 accounts, many posing as Spanish-language news outlets to share immigration-related misinformation. The accounts have been sharing doctored videos that contain artificial intelligence-generated content and the voices of well-known professional journalists to spread misinformation about immigration and other divisive topics that tend to go viral on social media, according to Alexios Mantzarlis, director of the Security, Trust, and Safety Initiative at Cornell Tech, Cornell University's graduate campus in New York City. Most of the TikTok accounts are named with generic terms that include the words "noticias" (news) or "noticiero" (newscast), and eight of them falsely use the logos of two major Spanish-language news networks, Univision and Telemundo. (Telemundo and NBC News are owned by NBCUniversal, a subsidiary of Comcast Corp.) This newest Spanish-language disinformation network "hijacks both the kind of viral mechanics of the TikTok algorithm, plus the very human peer-to-peer nature of immigrant communities," Mantzarlis, who also runs Faked Up, a newsletter about digital disinformation, told NBC News. Mantzarlis first started tracking the network of accounts last month after he saw a series of posts falsely claiming supermarkets would start requiring identification to buy groceries. In his newsletter, he said, some of the videos included messaging that pointed to "a clear effort to target undocumented immigrants." He noticed the accounts mainly focused on posting about President Donald Trump and immigration. This week, at least 25 videos shared by the network of TikTok accounts spreading false information about green card holders impersonated the voice of Univision reporter Javier Díaz. Díaz did not respond to a request for comment and Univision did not comment. The 25 videos were deleted from TikTok after NBC News sent links for the videos when requesting comment from the social media platform on Tuesday. NBC News also sent the company links for the 38 TikTok accounts Mantzarlis found were part of the network. By Wednesday afternoon, all 38 TikTok accounts were taken down. In a statement to NBC News, a TikTok spokesperson said, "We continue to protect our Spanish-speaking community from misleading content and accounts by investing in moderation teams and technologies, partnering with fact-checkers, and building media literacy tips and tools." According to the spokesperson, TikTok took down the videos and accounts because they violated the company's integrity and authenticity policies that prohibit harmful misinformation, misleading AI-generated content, impersonation and deceptive behaviors. A false narrative shared by one of the accounts in the network last month used the voice of former Telemundo anchor María Celeste Arrarás to make it look like she was reporting on Trump considering a plan to provide a legal pathway for undocumented immigrants without a criminal record. Arrarás posted the clip on her Instagram account to let her followers know the news in the video "IS FAKE." "They've edited several reports I presented on television years ago to mislead the public. IT'S AN AUDIO CUT to make it seem like I'm reporting news that ISN'T REAL. Please spread the word," she wrote in Spanish. As of Tuesday, that video featuring Arrarás' voice appeared to have already been deleted from the TikTok account. But the account remained functional until Wednesday morning along with several other videos containing misinformation, including a new one published the day before using Arrarás' voice. Before TikTok shut down the accounts Wednesday afternoon, the videos that included misinformation about green card holders and legal pathways for undocumented immigrants had reached the feed of an immigrant woman in New York City whom Amelia Scdoris, a community organizer at Cabrini Immigrant Services, had been helping. It's an example, Scdoris said, of why her organization holds regular meetings to help immigrants parse through misinformation they encounter and serve as a trustworthy source of immigration-related information. According to the TikTok spokesperson, the platform relies on automated technology, user reports, proactive searches, trend reports from experts and fact-checkers to detect misinformation on the platform across different languages. During the third quarter of 2024, over 97% of the content taken down from TikTok for violating its integrity and authenticity policies "was removed proactively before someone reported it to us." They added that TikTok has thousands of content moderators monitoring misinformation globally but declined to say exactly how many. As an example, TikTok's operation in the European Union region had 531 Spanish-language content moderators and 1,524 English-language content moderators as of December 2024, according to the company's most recent digital transparency report that it's required to file to operate in that region. Despite these efforts, combating online misinformation remains a daunting task. Latinos, including Spanish-speaking immigrant communities, are more likely to rely on social media to stay informed than other groups, according to a Nielsen report. TikTok is particularly popular among Latino communities in the U.S., with roughly half of Hispanic adults reporting they use the platform. Luba Cortés, the civil rights and lead immigration organizer at the advocacy group Make the Road New York, said she's seen firsthand how the news consumption habits of Spanish-speaking immigrant communities make them more vulnerable to the kind of misinformation spread on social media. "They see it and they believe that it's true," Cortés told NBC News about the AI-generated content and misinformation people are seeing online. "That causes a lot of mass panic." What's more, she said, the panic motivates viewers to share the false videos within their networks, since they mistakenly think the TikToks contain helpful information. Immigration-related falsehoods shared online often gain more traction when they include 'misleading cuts of evidence' connected to 'a real crisis,' according to Mert Bayar, who researches rumors about immigrants and noncitizens as a postdoctoral scholar at the University of Washington's Center for an Informed Public. That seems to be the case with the storyline falsely claiming green card holders, or lawful permanent residents, can't leave and re-enter the U.S. It's been circulating after news stories about green card holders now fighting deportation after participating in pro-Palestinian campus protests and traveling abroad. Last week, the National Korean American Service and Education Consortium, an immigration advocacy group, said in an Instagram post that some green card holders have unknowingly signed an I-407 form relinquishing their lawful permanent status. Cortés said she has not come across a case like that in her organization, but 'I do think, now, there are concerns.' Because the network of TikTok accounts sharing videos with false information both "denounce Trump's immigration policy and others that say he's about to save everyone," Mantzarlis said it's unclear if the operation is politically motivated or if it's laying the foundation to identify potential victims for future financial scams. As part of his previous research, Mantzarlis has seen how users who interact with this kind of content later become targets for monetary scams. "The only thing we can say for certain is that they're just trying to build a public, build a large audience," he said. "Afterwards, what they choose to do with that is kind of up to them." This article was originally published on
NBC News
26-03-2025
- Politics
- NBC News
How a TikTok network spreading Spanish-language immigration misinformation gained traction
A rumor falsely claiming green card holders are banned from leaving and re-entering the U.S. spread like wildfire on TikTok this week, garnering more than 21 million views across dozens of videos shared by a network of nearly 40 accounts, many posing as Spanish-language news outlets to share immigration-related misinformation. The accounts have been sharing doctored videos that contain artificial intelligence-generated content and the voices of well-known professional journalists to spread misinformation about immigration and other divisive topics that tend to go viral on social media, according to Alexios Mantzarlis, director of the Security, Trust, and Safety Initiative at Cornell Tech, Cornell University's graduate campus in New York City. Most of the TikTok accounts are named with generic terms that include the words "noticias" (news) or "noticiero" (newscast), and eight of them falsely use the logos of two major Spanish-language news networks, Univision and Telemundo. (Telemundo and NBC News are owned by NBCUniversal, a subsidiary of Comcast Corp.) This newest Spanish-language disinformation network "hijacks both the kind of viral mechanics of the TikTok algorithm, plus the very human peer-to-peer nature of immigrant communities," Mantzarlis, who also runs Faked Up, a newsletter about digital disinformation, told NBC News. Mantzarlis first started tracking the network of accounts last month after he saw a series of posts falsely claiming supermarkets would start requiring identification to buy groceries. In his newsletter, he said, some of the videos included messaging that pointed to " a clear effort to target undocumented immigrants." He noticed the accounts mainly focused on posting about President Donald Trump and immigration. This week, at least 25 videos shared by the network of TikTok accounts spreading false information about green card holders impersonated the voice of Univision reporter Javier Díaz. Díaz did not respond to a request for comment and Univision did not comment. The 25 videos were deleted from TikTok after NBC News sent links for the videos when requesting comment from the social media platform on Tuesday. NBC News also sent the company links for the 38 TikTok accounts Mantzarlis found were part of the network. By Wednesday afternoon, all 38 TikTok accounts were taken down. In a statement to NBC News, a TikTok spokesperson said, "We continue to protect our Spanish-speaking community from misleading content and accounts by investing in moderation teams and technologies, partnering with fact-checkers, and building media literacy tips and tools." According to the spokesperson, TikTok took down the videos and accounts because they violated the company's integrity and authenticity policies that prohibit harmful misinformation, misleading AI-generated content, impersonation and deceptive behaviors. A fake message — and messenger A false narrative shared by one of the accounts in the network last month used the voice of former Telemundo anchor María Celeste Arrarás to make it look like she was reporting on Trump considering a plan to provide a legal pathway for undocumented immigrants without a criminal record. Arrarás posted the clip on her Instagram account to let her followers know the news in the video " IS FAKE." "They've edited several reports I presented on television years ago to mislead the public. IT'S AN AUDIO CUT to make it seem like I'm reporting news that ISN'T REAL. Please spread the word," she wrote in Spanish. As of Tuesday, that video featuring Arrarás' voice appeared to have already been deleted from the TikTok account. But the account remained functional until Wednesday morning along with several other videos containing misinformation, including a new one published the day before using Arrarás' voice. Before TikTok shut down the accounts Wednesday afternoon, the videos that included misinformation about green card holders and legal pathways for undocumented immigrants had reached the feed of an immigrant woman in New York City whom Amelia Scdoris, a community organizer at Cabrini Immigrant Services, had been helping. It's an example, Scdoris said, of why her organization holds regular meetings to help immigrants parse through misinformation they encounter and serve as a trustworthy source of immigration-related information. According to the TikTok spokesperson, the platform relies on automated technology, user reports, proactive searches, trend reports from experts and fact-checkers to detect misinformation on the platform across different languages. During the third quarter of 2024, over 97% of the content taken down from TikTok for violating its integrity and authenticity policies "was removed proactively before someone reported it to us." They added that TikTok has thousands of content moderators monitoring misinformation globally but declined to say exactly how many. As an example, TikTok's operation in the European Union region had 531 Spanish-language content moderators and 1,524 English-language content moderators as of December 2024, according to the company's most recent digital transparency report that it's required to file to operate in that region. Despite these efforts, combating online misinformation remains a daunting task. Latinos, including Spanish-speaking immigrant communities, are more likely to rely on social media to stay informed than other groups, according to a Nielsen report. TikTok is particularly popular among Latino communities in the U.S., with roughly half of Hispanic adults reporting they use the platform. Luba Cortés, the civil rights and lead immigration organizer at the advocacy group Make the Road New York, said she's seen firsthand how the news consumption habits of Spanish-speaking immigrant communities make them more vulnerable to the kind of misinformation spread on social media. "They see it and they believe that it's true," Cortés told NBC News about the AI-generated content and misinformation people are seeing online. "That causes a lot of mass panic." What's more, she said, the panic motivates viewers to share the false videos within their networks, since they mistakenly think the TikToks contain helpful information. Immigration-related falsehoods shared online often gain more traction when they include 'misleading cuts of evidence' connected to 'a real crisis,' according to Mert Bayar, who researches rumors about immigrants and noncitizens as a postdoctoral scholar at the University of Washington's Center for an Informed Public. That seems to be the case with the storyline falsely claiming green card holders, or lawful permanent residents, can't leave and re-enter the U.S. It's been circulating after now fighting deportation after participating in pro-Palestinian campus protests and traveling abroad. Last week, the National Korean American Service and Education Consortium, an immigration advocacy group, said in an Instagram post that some green card holders have unknowingly signed an I-407 form relinquishing their lawful permanent status. Cortés said she has not come across a case like that in her organization, but 'I do think, now, there are concerns.' Politically motivated or a potential financial scam? Because the network of TikTok accounts sharing videos with false information both "denounce Trump's immigration policy and others that say he's about to save everyone," Mantzarlis said it's unclear if the operation is politically motivated or if it's laying the foundation to identify potential victims for future financial scams. As part of his previous research, Mantzarlis has seen how users who interact with this kind of content later become targets for monetary scams. "The only thing we can say for certain is that they're just trying to build a public, build a large audience," he said. "Afterwards, what they choose to do with that is kind of up to them."
