Cloudflare launches tool to help website owners monetise AI bot crawler access
has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday.
The tool allows website owners to choose whether
artificial intelligence
crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.
With AI crawlers increasingly collecting content without sending visitors to the original source, website owners are looking to develop additional revenue sources as search traffic referrals that once generated advertising revenue decline.
The initiative is supported by major publishers including Conde Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.
Cloudflare's Chief Strategy Officer Stephanie Cohen said the goal of such tools was to give publishers control over their content, and ensure a sustainable ecosystem for online content creators and
AI companies
.
"The change in traffic patterns has been rapid, and something needed to change," Cohen said in an interview. "This is just the beginning of a new model for the internet."
Google, for example, has seen its ratio of crawls to visitors referred back to sites drop to 18:1 from 6:1 just six months ago, according to Cloudflare data, suggesting the search giant is maintaining its crawling but decreasing referrals.
The decline could be a result of users finding answers directly within Google's search results, such as AI Overviews. Still, Google's ratio is much higher than other AI companies, such as OpenAI's 1,500:1.
For decades, search engines have indexed content on the internet directing users back to websites, an approach that rewards creators for producing quality content. However, AI companies' crawlers have disrupted this model because they harvest material without sending visitors to the original source and aggregate information through chatbots such as ChatGPT, depriving creators of revenue and recognition.
Many AI companies are circumventing a common web standard used by publishers to block the scraping of their content for use in AI systems, and argue they have broken no laws in accessing content for free.
In response, some publishers, including the New York Times, have sued AI companies for
copyright infringement
, while others have struck deals to license their content.
Reddit, for example, has sued AI startup Anthropic for allegedly scraping Reddit user comments to train its AI chatbot, while inking a
content licensing
deal with Google.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Time of India
42 minutes ago
- Time of India
'Dude clears interviews': Indian-origin entrepreneur says truth of Soham Parekh is simple
As Soham Parekh emerged as the 'Me Too' moment for US tech world, a US-based Indian-origin entrepreneur, Pritika Mehta, broke down the mystery and decided that the truth of Soham Parekh was pretty simple. Tired of too many ads? go ad free now Paresk cleared interviews as he was good with that, then he had junior people working for him like a small dev shop. The curious case of Soham Parekh broke the Internet after a US-based AI founder, Suhail Doshi, outed him on X claiming that Parekh has been moonlighting for years, which means he has been working for several software companies at the same time, and he is not even in the US. His resume went viral though his credentials are now disputed as he has been accused of fraud. 'Dog-eat-dog world out there' Another Indian-origin CEO Mihir Deo said Soham Parekh;s story did not surprise him at all. "That's just how it goes with a lot of the Indian engineering market, straight up. Some people are shocked by this, but if there's anyone who knows the Indian engineering market, the Soham Parekh stuff is like: "yeah, of course they do that"," he wrote. "I explored starting a business in India in the past. But the lack of trust that I had in people dissuaded me. People overpromise (and aren't afraid of overpromising) and often stab you in the back to get their lunch. It's a dog eat dog world out there. I decided against starting a company in India because I determined I wasn't thick skinned enough (e.g. also too rigorously honest as well) to play that game," Mihir Deo wrote. While US tech workers have been complaining about Indians eating up their jobs, the news of one Indian taking so many jobs in the US raised questions over the hiring and the management of these startups. Tech influencer Deedy Das suggested Parekh is 'just the tip of the iceberg,' pointing to a broader trend of 'overemployment' in remote work, with a Reddit community discussing similar cases.
Scroll.in
an hour ago
- Scroll.in
Can ChatGPT ‘rot' your brain as MIT study claims?
Since ChatGPT appeared almost three years ago, the impact of artificial intelligence technologies on learning has been widely debated. Are they handy tools for personalised education, or gateways to academic dishonesty? Most importantly, there has been concern that using AI will lead to a widespread 'dumbing down', or decline in the ability to think critically. If students use AI tools too early, the argument goes, they may not develop basic skills for critical thinking and problem-solving. Is that really the case? According to a recent study by scientists from MIT, it appears so. Using ChatGPT to help write essays, the researchers say, can lead to 'cognitive debt' and a 'likely decrease in learning skills'. So what did the study find? Brain vs AI Over the course of four months, the MIT team asked 54 adults to write a series of three essays using either AI (ChatGPT), a search engine, or their own brains ('brain-only' group). The team measured cognitive engagement by examining electrical activity in the brain and through linguistic analysis of the essays. The cognitive engagement of those who used AI was significantly lower than the other two groups. This group also had a harder time recalling quotes from their essays and felt a lower sense of ownership over them. Interestingly, participants switched roles for a final, fourth essay (the brain-only group used AI and vice versa). The AI-to-brain group performed worse and had engagement that was only slightly better than the other group's during their first session, far below the engagement of the brain-only group in their third session. The authors claim this demonstrates how prolonged use of AI led to participants accumulating 'cognitive debt'. When they finally had the opportunity to use their brains, they were unable to replicate the engagement or perform as well as the other two groups. Cautiously, the authors note that only 18 participants (six per condition) completed the fourth, final session. Therefore, the findings are preliminary and require further testing. Does AI really make us stupider? These results do not necessarily mean that students who used AI accumulated 'cognitive debt'. In our view, the findings are due to the particular design of the study. The change in neural connectivity of the brain-only group over the first three sessions was likely the result of becoming more familiar with the study task, a phenomenon known as the familiarisation effect. As study participants repeat the task, they become more familiar and efficient, and their cognitive strategy adapts accordingly. When the AI group finally got to 'use their brains', they were only doing the task once. As a result, they were unable to match the other group's experience. They achieved only slightly better engagement than the brain-only group during the first session. To fully justify the researchers' claims, the AI-to-brain participants would also need to complete three writing sessions without AI. Similarly, the fact the brain-to-AI group used ChatGPT more productively and strategically is likely due to the nature of the fourth writing task, which required writing an essay on one of the previous three topics. As writing without AI required more substantial engagement, they had a far better recall of what they had written in the past. Hence, they primarily used AI to search for new information and refine what they had previously written. What are the implications? To understand the current situation with AI, we can look back to what happened when calculators first became available. Back in the 1970s, their impact was regulated by making exams much harder. Instead of doing calculations by hand, students were expected to use calculators and spend their cognitive efforts on more complex tasks. Effectively, the bar was significantly raised, which made students work equally hard (if not harder) than before calculators were available. The challenge with AI is that, for the most part, educators have not raised the bar in a way that makes AI a necessary part of the process. Educators still require students to complete the same tasks and expect the same standard of work as they did five years ago. In such situations, AI can indeed be detrimental. Students can for the most part offload critical engagement with learning to AI, which results in ' metacognitive laziness '. However, just like calculators, AI can and should help us accomplish tasks that were previously impossible – and still require significant engagement. For example, we might ask teaching students to use AI to produce a detailed lesson plan, which will then be evaluated for quality and pedagogical soundness in an oral examination. In the MIT study, participants who used AI were producing the 'same old' essays. They adjusted their engagement to deliver the standard of work expected of them. The same would happen if students were asked to perform complex calculations with or without a calculator. The group doing calculations by hand would sweat, while those with calculators would barely blink an eye. Learning how to use AI Current and future generations need to be able to think critically and creatively and solve problems. However, AI is changing what these things mean. Producing essays with pen and paper is no longer a demonstration of critical thinking ability, just as doing long division is no longer a demonstration of numeracy. Knowing when, where and how to use AI is the key to long-term success and skill development. Prioritising which tasks can be offloaded to an AI to reduce cognitive debt is just as important as understanding which tasks require genuine creativity and critical thinking. Vitomir Kovanovic is Associate Professor and Associate Director of the Centre for Change and Complexity in Learning (C3L), Education Futures, University of South Australia. Rebecca Marrone.
Time of India
an hour ago
- Time of India
China's critical mineral squeeze: GTRI urges urgent localisation drive; reverse-engineering push seen key to cutting $100 bn trade gap
China's escalating export curbs on critical minerals are no longer just strategic warnings but a direct challenge to India's industrial ambitions, think tank Global Trade Research Initiative (GTRI) said on Thursday, urging immediate action through reverse-engineering and localisation of key imports to reduce dependency on Beijing. 'India must act to slash Chinese import dependence. There is an urgent need to do reverse-engineering of low- to mid-tech imports, domestic production incentives, and long-term investment in deep-tech manufacturing to reduce overreliance on a geopolitical rival and build economic resilience,' GTRI founder Ajay Srivastava said, PTI reported. GTRI noted that since mid-2023, China has restricted exports of gallium, germanium, and graphite — minerals essential for India's electronics, defence and clean energy sectors. The curbs have disrupted Indian industry's supply chains, with strategic ambiguity used to justify the restrictions under the guise of national security. Srivastava also flagged a recent incident where Chinese battery giant CATL allegedly instructed Foxconn to pull Chinese engineers from a plant near Chennai. 'The move disrupted timelines and coordination at a crucial time for India's electronics and EV supply chain buildout,' he said. by Taboola by Taboola Sponsored Links Sponsored Links Promoted Links Promoted Links You May Like Todos os idosos deveriam ter um desse no banheiro! Veja só! Evite quedas e escorregões Saiba Mais Undo As per GTRI, India's trade deficit with China surged to $100 billion in FY25, with imports rising even as exports fell. Chinese companies now dominate over 80% of India's supply in key areas such as laptops, solar panels, antibiotics, viscose yarn and lithium-ion batteries. To address this growing imbalance, GTRI recommended setting up sector-specific industrial labs to deconstruct high-volume imports and develop standardised open-access blueprints. These could be shared with MSMEs and large manufacturers to build domestic capacity. 'The first step is to launch a nationwide reverse-engineering initiative,' Srivastava said, adding that a 'Localize-100' tracker should be launched to monitor localisation of the top 100 imported items from China. Stay informed with the latest business news, updates on bank holidays and public holidays . AI Masterclass for Students. Upskill Young Ones Today!– Join Now
