Cloudflare launches tool to help website owners monetise AI bot crawler access
has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday.
The tool allows website owners to choose whether
artificial intelligence
crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.
With AI crawlers increasingly collecting content without sending visitors to the original source, website owners are looking to develop additional revenue sources as search traffic referrals that once generated advertising revenue decline.
The initiative is supported by major publishers including Conde Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.
Cloudflare's Chief Strategy Officer Stephanie Cohen said the goal of such tools was to give publishers control over their content, and ensure a sustainable ecosystem for online content creators and
AI companies
.
"The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Google, for example, has seen its ratio of crawls to visitors referred back to sites drop to 18:1 from 6:1 just six months ago, according to Cloudflare data, suggesting the search giant is maintaining its crawling but decreasing referrals.
The decline could be a result of users finding answers directly within Google's search results, such as AI Overviews. Still, Google's ratio is much higher than other AI companies, such as OpenAI's 1,500:1.
For decades, search engines have indexed content on the internet directing users back to websites, an approach that rewards creators for producing quality content. However, AI companies' crawlers have disrupted this model because they harvest material without sending visitors to the original source and aggregate information through chatbots such as ChatGPT, depriving creators of revenue and recognition.
Many AI companies are circumventing a common web standard used by publishers to block the scraping of their content for use in AI systems, and argue they have broken no laws in accessing content for free.
In response, some publishers, including the New York Times, have sued AI companies for
copyright infringement
, while others have struck deals to license their content.
Reddit, for example, has sued AI startup Anthropic for allegedly scraping Reddit user comments to train its AI chatbot, while inking a
content licensing
deal with Google.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
NDTV
42 minutes ago
- NDTV
Will The EU Delay Enforcing Its AI Act?
With less than a month to go before parts of the European Union's AI Act come into force, companies are calling for a pause in the provisions and getting support from some politicians. Groups representing big U.S. tech companies such as Google owner Alphabet and Facebook owner Meta, and European companies such as Mistral and ASML have urged the European Commission to delay the AI Act by years. What Is The August 2 Deadline? Under the landmark act that was passed a year earlier after intense debate between EU countries, its provisions would come into effect in a staggered manner over several years. Some important provisions, including rules for general purpose AI (GPAI) models, are due to apply on August 2. GPAI, which includes foundation models like those made by Google, Mistral and OpenAI, will be subject to transparency requirements such as drawing up technical documentation, complying with EU copyright law and providing detailed summaries about the content used for algorithm training. The companies will also need to test for bias, toxicity, and robustness before launching. AI models classed as posing a systemic risk and high-impact GPAI will have to conduct model evaluations, assess and mitigate risks, conduct adversarial testing, report to the European Commission on serious incidents and provide information on their energy efficiency. Why Do Companies Want A Pause? For AI companies, the enforcement of the act means additional costs for compliance. And for ones that make AI models, the requirements are tougher. But companies are also unsure how to comply with the rules as there are no guidelines yet. The AI Code of Practice, a guidance document to help AI developers to comply with the act, missed its publication date of May 2. "To address the uncertainty this situation is creating, we urge the Commission to propose a two-year 'clock-stop' on the AI Act before key obligations enter into force," said an open letter published on Thursday by a group of 45 European companies. It also called for simplification of the new rules. Another concern is that the act may stifle innovation, particularly in Europe where companies have smaller compliance teams than their U.S. counterparts. Will It Be Postponed? The European Commission has not yet commented on whether it will postpone the enforcement of the new rules in August. However, EU tech chief Henna Virkkunen promised on Wednesday to publish the AI Code of Practice before next month. Some political leaders, such as Swedish Prime Minister Ulf Kristersson, have also called the AI rules "confusing" and asked the EU to pause the act. "A bold 'stop-the-clock' intervention is urgently needed to give AI developers and deployers legal certainty, as long as necessary standards remain unavailable or delayed," tech lobbying group CCIA Europe said. The European Commission did not respond immediately to requests for comment.
India.com
3 hours ago
- India.com
Startup owner cancels Rs 22 lakh job offer due to candidate's religious comment on social media, says candidate is...
In a shocking turn of events that has taken the internet by storm, Mohammed Ahmed Bhati, an Indian startup founder of recruitment platform Jobbie, revoked a Rs 22 lakh-per-annum job offer to a candidate despite their impressive performance. As per media reports, Bhati rejected the candidate after finding a derogatory LinkedIn comments about religious communities during a routine background check. Here are all the details you need to know about the recent case. Why Ahmed Bhati rejected the candidate? 'No matter how talented someone is, respect and basic decency matter to us more. Talent gets you in the door, but values decide if you stay,' Bhati wrote in a widely circulated post, as reported by LiveMint. 'We were ready to extend an offer above our budget. But during the final background check, we found public posts that included remarks which could hurt religious sentiments. That's not a value we are willing to compromise on,' Bhati explained. Bhati said the candidate applied after coming across Jobbie's viral Reddit post detailing the company's 450 interviews for over 12,000 applicants. What set the individual apart was not only their technical skills but also their proactive approach—they built their resume using Jobbie's tools and even recommended improvements to the platform. What does the rejection note says? Bhati posted the rejection note which said, 'We came across some recent public posts on LinkedIn that included comments from you which will deeply hurt the religious sentiments of certain communities'. Social media users share reaction 'Revoking an offer letter for a professional position based on someone's social media activity goes out to show how this cancel culture can actually affect organisation when wrong people get in leadership position. And posting it here isn't virtue signalling either. It simply shows how immature and insecure you are as a leader. Good luck finding good candidates', an user shared his reaction to the whole scenario.
Mint
3 hours ago
- Mint
Pixel 6a to receive mandatory Android 16 update from July 8 amid battery safety fears
Google has officially announced a mandatory update to Android 16 for all Pixel 6a devices, as part of its newly launched Battery Performance Programme. The update, set to begin rolling out from 8 July, is aimed at addressing ongoing battery-related concerns which have plagued the handset in recent months. Until now, the Pixel 6a had been excluded from Android 16 eligibility, despite the update already being available across Pixel 6 to Pixel 9 models. However, Google is now making the update compulsory for Pixel 6a users in order to activate new safety measures that help mitigate potential risks associated with battery performance. According to a statement published on Google's support page, the update will specifically target devices flagged as 'Impacted Devices'. These units will receive the software update automatically. Once installed, the update will enable advanced battery management features designed to prevent overheating and possible damage. One of the major changes coming with the update is a limit on battery capacity and charging performance once the device reaches 400 full charge cycles. However, until this milestone is hit, the changes will remain dormant. Users of impacted devices will receive an alert once their handset approaches 375 charge cycles, reminding them of the impending changes. Post-update, they may notice slower charging speeds and temporary fluctuations in the battery-level indicator as the phone adjusts to the new parameters. The initiative follows a series of reported incidents involving the Pixel 6a. In one case earlier this year, a user shared images of a swollen battery causing the screen to lift from the frame. A few months later, a separate report emerged alleging that a Pixel 6a had exploded due to battery expansion. In light of these issues, Google is offering free repairs for devices enrolled in the Battery Performance Programme. Alternatively, affected users can opt for a cash reimbursement or a discount voucher to use on future hardware purchases through the Google Store.
