Cloudflare blocks AI crawlers by default in a major win for web publishers
Historical context: How it used to work
Traditionally, the web operated on a 'content-for-traffic' model: search engines crawled sites and sent visitors back via links. This model supported ad revenue and visibility. With the rise of generative AI, firms have harvested vast amounts of web content like text, images, code, without driving any traffic back to publishers.
What Cloudflare is doing now
New Cloudflare users are now opted‑out of AI scraper access by default. Owners must explicitly choose to allow bots. An optional Pay Per Crawl program is also in development: publishers can monetize access by charging AI companies per crawl. Major players like Time, Reddit, BuzzFeed, and The Atlantic have already expressed support.
How this affects AI companies
AI firms relying on large language models such as OpenAI, Google, Meta, and Anthropic will face new hurdles. They must now seek permission or negotiate access fees, reducing free access to diverse data. The rise of 'AI Labyrinth,' a honeypot of decoy pages, further complicates unregulated scraping.
Technical and operational roadblocks
AI scrapers pose challenges. They ignore existing protections like robots.txt, cause server strain, and increase bandwidth costs. Cloudflare's answer includes advanced bot detection powered by behavioural analysis and ML, plus the labyrinth tactic to confuse scrapers.
Industry reactions and implications
Content owners, including Universal Music Group and Reddit, have welcomed the change, calling it essential to sustainable web economics. Cloudflare's CEO Matthew Prince summed it up: 'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve'. But critics worry that a permission-based web could limit AI innovation.
New web marketplace on the horizon
Cloudflare is exploring a content-access marketplace where AI firms pay for usage, transforming free scrapes into transactions. This could value content by data rather than clicks, offering publishers a fresh revenue channel.
Cloudflare's move shifts control back to creators, forcing a rethink of how AI companies gather data. Publishers now have the power to block, charge, or allow bots. This shift may define the economic and ethical foundations of the AI-powered web.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Time of India
15 minutes ago
- Time of India
Virgo Daily Health Horoscope Today, July 15, 2025: Embrace the wisdom within struggle
Challenges might seem frustrating now, but within them are lessons that will serve you for a long time. You may feel things are moving slower than expected, but every delay is guiding you to better clarity. Don't rush the process. Trust that what you're going through now is shaping you into a wiser version of yourself. Virgo Health Horoscope Today Your health may need more attention to routine today. Stick to your usual eating and sleeping habits as much as possible. If digestion has been irregular or you've been skipping meals, it's time to correct that. Include grounding foods like khichdi or warm soup. Gentle stretches and keeping yourself hydrated will go a long way in maintaining your strength and balance today. Virgo Wellness Horoscope Today Your mind may feel scattered with overthinking. Try not to chase every thought. Slow down and return to the present moment. Doing something with your hands—like drawing, gardening or cooking—can help bring mental calm. Let go of the need for perfection in everything. Wellness grows when you allow life to be simple and yourself to be human. Virgo Love Horoscope Today Love may ask for patience today. If you are single, don't lose hope because something didn't work out in the past. by Taboola by Taboola Sponsored Links Sponsored Links Promoted Links Promoted Links You May Like Is it legal? How to get Internet without paying a subscription? Techno Mag Learn More Undo A meaningful connection is on the way, but it requires you to heal first. If you are with someone, focus on listening without judging. Sometimes love is not about fixing but simply being there with presence and care. Virgo Career Horoscope Today Your work life may feel heavy with responsibilities, but you are more organised than you think. Prioritise one task at a time instead of trying to do everything. A senior may notice your consistency. Keep your communication honest and your intentions clear. This is not the time to seek praise, but rather to build strong foundations for the future. Virgo Money Horoscope Today Money matters look steady today. A past investment or idea may begin to show small results. If you are thinking of saving or planning ahead, today is perfect for creating a practical strategy. You don't have to wait for everything to be perfect—start with what you have. Prosperity will follow step by step, as long as your actions remain focused and sincere. Virgo Affirmation Today: Every challenge helps me grow and evolve peacefully. Discover everything about astrology at the Times of India , including daily horoscopes for Aries , Taurus , Gemini , Cancer , Leo , Virgo , Libra , Scorpio , Sagittarius , Capricorn , Aquarius , and Pisces .
Mint
an hour ago
- Mint
AI is killing the web. Can anything save it?
Around the beginning of last year, Matthew Prince started receiving worried calls from the chief executives of large media companies. They told Mr Prince, whose firm, Cloudflare, provides security infrastructure to about a fifth of the web, that their businesses faced a grave new online threat. 'I said, 'What, is it the North Koreans?'," he recalls. 'And they said, 'No. It's AI'." Those executives had spotted the early signs of a trend that has since become clear: artificial intelligence is transforming the way that people navigate the web. As users pose their queries to chatbots rather than conventional search engines, they are given answers, rather than links to follow. The result is that 'content" publishers, from news providers and online forums to reference sites such as Wikipedia, are seeing alarming drops in their traffic. As AI changes how people browse, it is altering the economic bargain at the heart of the internet. Human traffic has long been monetised using online advertising; now that traffic is drying up. Content producers are urgently trying to find new ways to make AI companies pay them for information. If they cannot, the open web may evolve into something very different. Since the launch of ChatGPT in late 2022, people have embraced a new way to seek information online. OpenAI, maker of ChatGPT, says that around 800m people use the chatbot. It is the most popular download on the iPhone app store. Apple said that conventional searches in its Safari web browser had fallen for the first time in April, as people posed their questions to AI instead. OpenAI is soon expected to launch a browser of its own. Its rise is so dramatic that a Hollywood adaptation is in the works. As OpenAI and other upstarts have soared, Google, which has about 90% of the conventional search market in America, has added AI features to its own search engine in a bid to keep up. Last year it began preceding some search results with AI-generated 'overviews", which have since become ubiquitous. In May it launched 'AI mode", a chatbot-like version of its search engine. The company promises that, with AI, users can 'let Google do the Googling for you". Yet as Google does the Googling, humans no longer visit the websites from which the information is gleaned. Similarweb, which measures traffic to more than 100m web domains, estimates that worldwide search traffic (by humans) fell by about 15% in the year to June. Although some categories, such as hobbyists' sites, are doing fine, others have been hit hard (see chart). Many of the most affected are just the kind that might have commonly answered search queries. Science and education sites have lost 10% of their visitors. Reference sites have lost 15%. Health sites have lost 31%. For companies that sell advertising or subscriptions, lost visitors means lost revenue. 'We had a very positive relationship with Google for a long time…They broke the deal," says Neil Vogel, head of Dotdash Meredith, which owns titles such as People and Food & Wine. Three years ago its sites got more than 60% of their traffic from Google. Now the figure is in the mid-30s. 'They are stealing our content to compete with us," says Mr Vogel. Google has insisted that its use of others' content is fair. But since it launched its AI overviews, the share of news-related searches resulting in no onward clicks has risen from 56% to 69%, estimates Similarweb. In other words, seven in ten people get their answer without visiting the page that supplied it. 'The nature of the internet has completely changed," says Prashanth Chandrasekar, chief executive of Stack Overflow, best known as an online forum for coders. 'AI is basically choking off traffic to most content sites," he says. With fewer visitors, Stack Overflow is seeing fewer questions posted on its message boards. Wikipedia, also powered by enthusiasts, warns that AI-generated summaries without attribution 'block pathways for people to access…and contribute to" the site. To keep the traffic and the money coming, many big content producers have negotiated licensing deals with AI companies, backed up by legal threats: what Robert Thomson, chief executive of News Corp, has dubbed 'wooing and suing".His company, which owns theWall Street Journalandthe New York Post, among other titles, has struck a deal with OpenAI. Two of its subsidiaries are suing Perplexity, another AI answer engine. TheNew York Timeshas done a deal with Amazon while suing OpenAI. Plenty of other transactions and lawsuits are going on. (The Economist's parent company has not taken a public position on whether it will license our work.) Yet this approach has limits. For one thing, judges so far seem minded to side with AI companies: last month two separate copyright cases in California went in favour of their defendants, Meta and Anthropic, both of which argued that training their models on others' content amounted to fair use. President Donald Trump seems to acceptSilicon Valley's argument that it must be allowed to get on with developing the technology of the future before China can. He has appointed tech boosters as advisers on AI, and sacked the head of the US Copyright Office soon after she argued that training AI on copyrighted material was not always legal. AI companies are more willing to pay for continuing access to information than training data. But the deals done so far are hardly stellar. Reddit, an online forum, has licensed its user-generated content to Google for a reported $60m a year. Yet its market value fell by more than half—over $20bn—after it reported slower user-growth than expected in February, owing to wobbles in search traffic. (Growth has since picked up and Reddit's share price has recovered some lost ground.) The bigger problem, however, is that most of the internet's hundreds of millions of domains are too small to either woo or sue the tech giants. Their content may be collectively essential to AI firms, but each site is individually dispensable. Even if they could join forces to bargain collectively, antitrust law would forbid it. They could block AI crawlers, and some do. But that means no search visibility at all. Software providers may be able to help. All of Cloudflare's new customers will now be asked if they want to allow AI companies' bots to scrape their site, and for what purpose. Cloudflare's scale gives it a better chance than most of enabling something like a collective response by content sites that want to force AI firms to cough up. It is testing a pay-as-you-crawl system that would let sites charge bots an entry fee. 'We have to set the rules of the road," says Mr Prince, who says his preferred outcome is 'a world where humans get content for free, and bots pay a tonne for it". An alternative is offered by Tollbit, which bills itself as a paywall for bots. It allows content sites to charge AI crawlers varying rates: for instance, a magazine could charge more for new stories than old ones. In the first quarter of this year Tollbit processed 15m micro-transactions of this sort, for 2,000 content producers including the Associated Pressand Newsweek. Toshit Panigrahi, its chief executive, points out that whereas traditional search engines incentivise samey content—'What time does the Super Bowl start?", for example—charging for access incentivises uniqueness. One of Tollbit's highest per-crawl rates is charged by a local newspaper. Another model is being put forward by ProRata, a startup led by Bill Gross, a pioneer in the 1990s of the pay-as-you-click online ads that have powered much of the web ever since. He proposes that money from ads placed alongside AI-generated answers should be redistributed to sites in proportion to how much their content contributed to the answer. ProRata has its own answer engine, which shares ad revenue with its 500-plus partners, which include theFinancial Timesand theAtlantic. It is currently more of an exemplar than a serious threat to Google: Mr Gross says his main aim is to 'show a fair business model that other people eventually copy". Meanwhile, content producers are rethinking their business models. 'The future of the internet is not all about traffic," says Mr Chandrasekar, who has built up Stack Overflow's private, enterprise-oriented subscription product, Stack Internal. News publishers are planning for 'Google zero", deploying newsletters and apps to reach customers who no longer come to them via search, and moving their content behind paywalls or to live events. Audio and video are proving legally and technically harder for AI engines to summarise than text. The site to which answer engines refer search traffic most often, by far, is YouTube, according to Similarweb. Not everyone thinks the web is in decline—on the contrary, it is in 'an incredibly expansionary moment", argues Robby Stein of Google. As AI makes it easier to create content, the number of sites is growing: Google's bots report that the web has expanded by 45% in the past two years. AI search lets people ask questions in new ways—for instance, taking a photo of their bookshelf and asking for recommendations on what to read next—which could increase traffic. With AI queries, more sites than ever are being 'read", even if not with human eyes. An answer engine may scan hundreds of pages to deliver an answer, drawing on a more diverse range of sources than human readers would. As for the idea that Google is disseminating less human traffic than before, Mr Stein says the company has not noticed a dramatic decline in the number of outbound clicks, though it declines to make the number public. There are other reasons besides AI why people may be visiting sites less. Maybe they are scrolling social media. Maybe they are listening to podcasts. The death of the web has been predicted before—at the hands of social networks, then smartphone apps—and not come to pass. But AI may pose the biggest threat to it yet. If the web is to continue in something close to its current form, sites will have to find new ways to get paid for content. 'There's no question that people prefer AI search," says Mr Gross. 'And to make the internet survive, to make democracy survive, to make content creators survive, AI search has to share revenue with creators."
United News of India
an hour ago
- United News of India
Reddit starts verifying ages of users in UK
London, July 15 (UNI) Reddit has announced it is introducing age verification technique on its UK site to stop people aged under 18 from looking at "certain mature content". The social media platform is bringing in the measures to comply with new rules under the UK's Online Safety Act which require sites that show adult material to introduce "robust" age checking techniques. Reddit, known for its online communities and discussions, said that while it does not want to know who its audience is: "It would be helpful for our safety efforts to be able to confirm whether you are a child or an adult." Ofcom, the UK regulator, said: "We expect other companies to follow suit, or face enforcement if they fail to act." Reddit said that from July 14 (Monday), an outside firm called Persona will perform age verification for the social media platform either through an uploaded selfie or "a photo of your government ID", such as a passport, reports BBC. It said Reddit will not have access to the photo and will only retain a user's verification status and date of birth, so people do not have to re-enter it each time they try to access restricted content. Reddit added that Persona "promises not to retain the picture for longer than seven days" and will not have access to a user's data on the site. The new rules in the UK come into force on July 25. A spokesperson for Ofcom said: "Society has long protected youngsters from products that aren't suitable for them, from alcohol to smoking or gambling. "Now, children will be better protected from online material that's not appropriate for them, while adults' rights to access legal content are preserved." But Scott Babwah Brennen, director of the Center on Technology Policy at New York University, said: "There's always going to be ways that kids can get around it and there's always going to be concerns about who's collecting personal or sensitive information, and how long are they holding it." Pornhub and a number of other major adult websites recently confirmed they would introduce enhanced age checks in time for the new rules. Pornhub's parent company, Aylo, said it would bring in "government approved age assurance methods" but is yet to reveal how it will require users to prove they are over 18. Ofcom has previously said simply clicking a button, which is all the adult site currently requires, is not enough. Companies that fail to meet the rules face fines of up to £18m or 10% of worldwide revenue, "whichever is greater". It added that in the most serious cases, it can seek a court order for "business disruption measures", such as requiring payment providers or advertisers to withdraw their services from a platform, or requiring Internet Service Providers to block access to a site in the UK." UNI XC SS
