Cloudflare launches tool to help website owners monetise AI bot crawler access
The tool allows website owners to choose whether artificial intelligence crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.
With AI crawlers increasingly collecting content without sending visitors to the original source, website owners are looking to develop additional revenue sources as search traffic referrals that once generated advertising revenue decline.
The initiative is supported by major publishers including Condé Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.
Cloudflare's Chief Strategy Officer Stephanie Cohen said the goal of such tools was to give publishers control over their content, and ensure a sustainable ecosystem for online content creators and AI companies.
"The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Google, for example, has seen its ratio of crawls to visitors referred back to sites drop to 18:1 from 6:1 just six months ago, according to Cloudflare data, suggesting the search giant is maintaining its crawling but decreasing referrals.
The decline could be a result of users finding answers directly within Google's search results, such as AI Overviews. Still, Google's ratio is much higher than other AI companies, such as OpenAI's 1,500:1.
For decades, search engines have indexed content on the internet directing users back to websites, an approach that rewards creators for producing quality content. However, AI companies' crawlers have disrupted this model because they harvest material without sending visitors to the original source and aggregate information through chatbots such as ChatGPT, depriving creators of revenue and recognition.
Many AI companies are circumventing a common web standard used by publishers to block the scraping of their content for use in AI systems, and argue they have broken no laws in accessing content for free.
In response, some publishers, including the New York Times, have sued AI companies for copyright infringement, while others have struck deals to license their content.
Reddit, for example, has sued AI startup Anthropic for allegedly scraping Reddit user comments to train its AI chatbot, while inking a content licensing deal with Google.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Hans India
35 minutes ago
- Hans India
Google AI mode debuts in India: A ChatGPT-like search experience powered by Gemini
A quiet revolution just landed in your Google Search bar. Google AI Mode, first unveiled at Google I/O 2025, is now live in India through the Google app on Android and iOS. Originally available only in the U.S., this marks the first major international rollout of Google's conversational search upgrade. Launched with a celebratory animation on the Google homepage on July 1st, AI Mode signals a major leap forward in how we interact with search engines. Rather than relying solely on keyword-based results, AI Mode delivers an interactive, back-and-forth experience powered by a customized version of the Gemini AI model. This new feature can handle complex, multi-layered questions and respond with detailed, natural language answers—often backed by cited sources. Whether you're looking for AI tool recommendations, planning a vacation, or comparing shopping options, AI Mode aims to make search more intelligent and personalized. 'AI Mode is where we will first bring our frontier capabilities into Search,' said Sundar Pichai, CEO of Google, during the I/O keynote. How AI Mode Works Available via the Search Labs section in the Google app, AI Mode appears as an optional tab. Once activated, it transforms search into a conversational space. Using a technique called 'query fan-out,' it breaks your input into multiple layers of intent, offering more comprehensive responses. The feature is multi-modal, allowing users to input queries via voice, text, or images—making it one of Google's most dynamic tools to date. Key Features Rolling Out in India: Deep Search: Delivers well-reasoned, cited answers to complex questions. Shopping Tools: Enables product comparison and virtual clothing try-on. Data Visualization: Generates real-time graphs for finance, weather, or sports. App Integration: With your permission, it can personalize suggestions using Gmail or Calendar data—such as recommending eateries near your next meeting. Why Now? AI Mode builds on Google's earlier AI Overviews, which offered generative summaries in standard search. But this goes further—creating an intelligent, interactive zone for discovery, planning, and exploration. 'This is the future of Google Search—a search that goes beyond information to intelligence,' said Elizabeth Reid, VP and Head of Search at Google. Over time, Google plans to incorporate many of AI Mode's capabilities into its core Search experience. Is It for Everyone? Not necessarily. Experts say AI Mode is ideal for nuanced, research-heavy queries. Simple fact-checks or quick lookups might still be faster with classic search. 'The percentage of people who want AI Mode for everything will be surprisingly low,' said Eugene Levin, president of Semrush, in an interview with CNET. Still, the option is there—and it's evolving fast. Whether AI Mode becomes the default or remains a tool for deeper queries, one thing is clear: the future of search is no longer just about links, but about context, reasoning, and dialogue.
Time of India
an hour ago
- Time of India
14 million Android users in California to get $314.6m from Google. Check reason
A California jury has ruled against Google, ordering the tech giant to pay over $314.6 million to Android users. The lawsuit, representing 14 million users, alleged Google collected data from idle devices without permission, impacting users' data plans. Google disputes the verdict, claiming the data transfers are essential for device security and performance, and plans to appeal the decision. Tired of too many ads? Remove Ads Google asked to pay $314.6 million Tired of too many ads? Remove Ads A jury in San Jose, California, said on Tuesday that Google misused customers' cellphone data and must pay more than $314.6m to Android smartphone users in the state, according to an attorney for the plaintiffs. The verdict found Google liable for collecting information from idle Android devices without user permission, sending and receiving data even when phones were inactive. Google has said it did nothing wrong and will jury sided with plaintiffs representing an estimated 14 million California Android users in a class action lawsuit filed in 2019, causing what the lawsuit had called "mandatory and unavoidable burdens shouldered by Android device users for Google's benefit."The class action lawsuit was brought on behalf of an estimated 14 million California Android users, first filed in 2019. Plaintiffs argued that Google's actions imposed "mandatory and unavoidable burdens" for its own benefit, primarily using the data for targeted advertising and consuming users' cellular data at their plaintiffs' legal team argued that phone users' data is their personal property under California law and that Google is thus liable for data used while customers' phones were idle dating back to 2016. The plaintiffs' attorney Glen Summers said the verdict "forcefully vindicates the merits of this case and reflects the seriousness of Google's misconduct."Google spokesperson Jose Castaneda said in a statement that the company would appeal, and that the verdict "misunderstands services that are critical to the security, performance, and reliability of Android devices."Google spokesperson Jose Castaneda told Newsweek: "We strongly disagree with today's decision and will appeal. This ruling is a setback for users, as it misunderstands services that are critical to the security, performance, and reliability of Android devices."Google told the court that no Android users were harmed by the data transfers and that users consented to them in the company's terms of service and privacy group filed a separate lawsuit in federal court in San Jose, bringing the same claims against Google on behalf of Android users in the other 49 states. That case is scheduled for trial in April is one of several major tech companies expected to reduce its headcount in July, according to Worker Adjustment and Retraining Notification Act notices, which must be issued prior to layoffs. According to the Los Angeles Times, Google was one of a number of Bay Area companies to lay off employees in the first quarter of the year.(With inputs from Reuters)
Economic Times
an hour ago
- Economic Times
July fourth cookouts face record beef prices amid shrinking cattle supply
TIL Creatives Ground beef prices have reached a record high of $6.67 per pound, reflecting historic lows in U.S. cattle inventories and rising demand (AI generated image) Ground beef prices have hit an all-time high, with retail costs for 2 pounds averaging $13.33, or $6.67 per pound, according to the American Farm Bureau Federation (AFBF). That's the highest price since the group began tracking in 2013. While consumers may find slight relief in the total cost of a traditional cookout, $70.92 for 10 people, or $7.09 per person, that figure is still 19% higher than it was five years ago. And beef is largely to spike in prices comes despite steady demand. 'Consumer demand for quality beef is not declining,' said Samantha Ayoub, associate economist at the AFBF and author of the new survey. 'If anything, it's proving to be incredibly resilient.'Cattle supply hits historic lows The root of the price surge is a shrinking cattle supply. The U.S. Department of Agriculture (USDA) reports that the national cattle herd stands at 86.7 million, the lowest since 1951. Prolonged drought conditions in key grazing regions have worsened the situation, depleting pastureland and enabling the spread of drought-tolerant weeds that are nutritionally poor, or even toxic, for livestock. Compounding the pressure is concern over the spread of the New World screwworm, a parasitic fly species that lays eggs in open wounds of animals. The larvae burrow into flesh, causing severe infections. Although no cases have been reported in the US, recent infestations have reached as far north as Oaxaca, Mexico. Fearing potential outbreaks, the USDA in May suspended cattle imports from Mexico, a critical source of feeder cattle for the US beef market.'This pest poses a serious risk not only to Texas livestock and wildlife but also to the economic backbone of our rural communities,' said Texas Agriculture Commissioner Sid futures reflect investor sentiment The cattle market has responded accordingly. Live cattle futures on the Chicago Mercantile Exchange have surged over 10% this year, hitting a record $2.19 per pound on June 9, making it one of the best-performing commodities in 2025. Other proteins offer a breakFortunately for consumers, not all meat prices are rising as sharply. Despite challenges from avian flu, the average cost of 2 pounds of chicken breast rose just 4 cents to $7.79. Pork prices actually declined, with 3 pounds of pork chops falling 9% to $14.13, according to the the soaring cost of ground beef underscores the broader strain in the nation's livestock sector, one that could reshape future pricing and availability unless weather patterns shift or import channels reopen. Until then, that Independence Day burger is likely to be a premium indulgence.
