Latest news with #AIscrapers
New York Times
01-08-2025
- Entertainment
- New York Times
Age-Gating the Internet + Cloudflare Takes On A.I. Scrapers + HatGPT
Hosted by Kevin Roose and Casey Newton Produced by Rachel Cohn and Whitney Jones Edited by Jen Poyant Engineered by Chris Wood Original music by Dan PowellElisheba IttoopRowan Niemisto and Alyssa Moxley This week, we look at the fallout from a sweeping internet age-verification law that went into effect in Britain. We explain why age restrictions are suddenly popping up all over the internet — and how some might create more problems than they solve. Then Matthew Prince, chief executive of Cloudflare, returns to the show to discuss his company's new plan to help publishers fight back against A.I. scrapers and potentially to create a new online marketplace for quality content in the process. Finally, we round up some headlines from around the tech world in the latest round of HatGPT. Guests: Matthew Prince, chief executive of Cloudflare Additional Reading: Supreme Court Upholds Texas Law Limiting Access to Pornography The U.K.'s age gates are coming to America Cloudflare Introduces Default Blocking of A.I. Data Scrapers Also, you can still get a special-edition 'Hard Fork' hat! For a limited time, you'll receive one when you purchase an annual New York Times Audio subscription for the first time. Go to 'Hard Fork' is hosted by Kevin Roose and Casey Newton, and produced by Rachel Cohn and Whitney Jones. We're edited by Jen Poyant. Engineering by Chris Wood and original music by Dan Powell, Elisheba Ittoop, Alyssa Moxley and Rowan Niemisto. Fact-checking by Caitlin Love. Special thanks to Paula Szuchman, Pui-Wing Tam, Dahlia Haddad and Jeffrey Miranda.
The Verge
01-07-2025
- Business
- The Verge
Cloudflare will now block AI crawlers by default
The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from 'accessing content without permission or compensation,' according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a 'Pay Per Crawl' fee. The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content. AI companies can then view pricing and choose whether to register for the 'Pay Per Crawl' fee or turn away. This is only available for 'a group of some of the leading publishers and content creators' for now, but Cloudflare says it will ensure 'AI companies can use quality content the right way — with permission and compensation.' Cloudflare has been helping domain owners fight AI crawlers for a while now. The company started letting websites block AI crawlers in 2023, but it only applied to ones that abide by a site's file, the unenforceable agreement that signals whether bots can scrape its content. Cloudflare began allowing websites to block 'all' AI bots last year — whether they respect a site's file or not — and now this setting is enabled by default for new Cloudflare customers. (The company identifies scrapers to block by comparing them to its list of known AI bots.) Cloudflare also rolled out a feature in March that sends web-crawling bots into an 'AI Labyrinth' to deter them from scraping sites without permission. Several major publishers and online platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, are on board with Cloudflare's new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. 'People trust the AI more over the last six months, which means they're not reading original content,' Cloudflare CEO Matthew Prince said during the Axios Live event last week. Additionally, Cloudflare says it's working with AI companies to help verify their crawlers and allow them to 'clearly state their purpose,' such as whether they're using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in. 'Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,' Prince said in the press release. 'AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.'
