AI audits and 'pay per crawl': How Cloudflare is trying to fix a 'broken' web model
Cloudflare said publishers using the cloud company's tools for hosting websites will now block AI crawlers by default from accessing and poaching content without permission.
'Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access,' the company said.
'This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out.'
Cloudflare first addressed concerns about AI data-scraping last year when it gave websites the option to block AI companies from poaching content.
'Now by default you have control over who crawls your site and what that information is used for,' said Stephanie Cohen, a chief strategy officer for Cloudflare.
'The benefit of that is that it creates the conditions for a new business model of the internet to develop,' she told The National after Cloudflare introduced the new settings and options.
Some of the strongest proponents of AI tools, and many of the tools' creators, have justified data scraping, saying it is akin to the early days of search engines when controversy briefly surfaced over whether or not search companies should be able to index sites.
Others say that comparison isn't appropriate, because search engines didn't poach the contents of entire websites.
Additionally, during the early days of web browsers, search engines and the crawlers they implemented provided a framework that built much of the internet as we know it.
It was a win-win situation for the likes of Google and media companies which provided information and sought to attract audiences by delivering web traffic through internet searches.
The debut of OpenAI's ChatGPT in 2022 and other AI platforms turned that economic model on its head.
Instead of directing traffic to websites, AI summaries have quickly become a destination unto themselves, siphoning traffic from the same websites from which they scrape data.
Ms Cohen said publishers and content creators using Cloudflare's services soon noticed a major dip in web traffic.
'Not only was it getting more difficult to get web traffic – to the tune of it being 10 times harder – but it was also getting more difficult at a faster and faster rate,' she said.
The web's economics based on search that built up over the last decade, she said, started to erode over a period of six months.
In 2024, Ms Cohen said Cloudflare allowed users to see which AI companies were scraping their sites and turn off that ability. This year they are taking things further by introducing 'pay per crawl'.
The tool gives publishers and website operators the option of allowing AI scraping for free, charging for it 'at the configured domain-wide price,' or blocking scraping entirely.
As AI developments quickens, so too does the bad blood between media organisations and the tech firms driving the AI boom.
Several lawsuits have been filed. The New York Times has sued OpenAI and Microsoft for allegedly using its articles to power increasingly popular chatbots.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
UAE Moments
5 hours ago
- UAE Moments
♍ Virgo: Daily Horoscope for July 12th, 2025
Dear Virgo, today's celestial dance invites you to embrace the beauty of order and detail. As you navigate your day, remember the spiritual advice: "In the tapestry of life, every thread has its purpose." Love Horoscope Today, Virgo, the celestial energies encourage you to open your heart and express your feelings more freely. While your natural tendency might be to analyze and critique, try to embrace the spontaneity of love. Whether you're in a relationship or single, let your guard down a bit and allow yourself to be vulnerable. Share a laugh, a story, or a dream with someone special. Remember, love isn't about perfection; it's about connection. So, let go of the need to control every detail and enjoy the beautiful chaos of human emotions. Career Horoscope At work, Virgo, your analytical skills are in high demand today. The Moon's opposition to Mercury suggests that communication may be a bit challenging, but your logical approach can help bridge any gaps. Take the time to listen actively and express your ideas clearly. Your attention to detail will be appreciated by colleagues and superiors alike. Remember, teamwork is key, so be open to collaboration and new ideas. Your ability to stay organized and focused will ensure that you navigate any challenges with grace and efficiency. Health Horoscope Your well-being today is closely tied to your ability to balance your meticulous nature with a bit of relaxation. The Moon's influence suggests that taking a break from your usual routine can do wonders for your mental health. Consider engaging in a creative activity that allows you to express yourself without judgment, like painting or dancing. Remember, Virgo, it's okay to let loose and have fun. Your mind and body will thank you for it, and you'll return to your tasks with renewed energy and clarity. Finance Horoscope Financially, Virgo, today is a day to focus on the details. The planetary alignments suggest that you might uncover a small but significant opportunity to improve your financial situation. Whether it's reviewing your budget, finding a better deal, or spotting an investment opportunity, your keen eye for detail will serve you well. Trust in your ability to make sound financial decisions, but also be open to advice from trusted sources. Remember, small changes can lead to significant improvements over time.
TECHx
12 hours ago
- TECHx
Khazna Data Signs Deal to Build AI Data Center in Italy
Home » Tech Value Chain » Global Brands » Khazna Data Signs Deal to Build AI Data Center in Italy Khazna Data Centers, hyperscale digital infrastructure, and Italian energy giant Eni have announced the signing of a Head of Terms (HoT) to form a Joint Venture. The agreement will support the development of a 500 MW AI Data Center Campus in Ferrera Erbognone, Lombardy. The signing ceremony took place in Rome in the presence of Her Excellency Reem Al Hashimy, UAE Minister of State for International Cooperation. Senior UAE officials and Italy's Minister of Enterprises and Made in Italy, Adolfo Urso, also attended the event. Khazna CEO Hassan Alnaqbi and Eni COO Global Natural Resources Guido Brusco signed the agreement. The collaboration brings together Khazna's advanced data center design capabilities and Eni's expertise in sustainable energy. The new AI Data Center Campus will be powered by 'Blue Power' supplied by Eni. This low-carbon electricity is generated by a high-efficiency Combined Cycle Gas Turbine (CCGT) Power Plant equipped to capture CO2 emissions. The project introduces a first-of-its-kind synergy between decarbonized energy and data center design in Italy. This agreement marks a key milestone in a broader program to establish up to 1 GW of total IT capacity in Italy. The program stems from a Letter of Intent signed during the state visit of His Highness Sheikh Mohamed bin Zayed Al Nahyan, President of the UAE, to Italy in February 2025. The visit emphasized strategic cooperation between the UAE and Italy in AI, digital infrastructure, and the energy transition. The HoT confirms Khazna's role as the primary partner for designing and developing a phased master plan to reach the 500 MW goal. The Milan-based campus will support high-density AI and high-performance computing workloads. Key highlights: • The project enables sustainable, scalable AI infrastructure in Europe. • It reflects strategic alignment between UAE and Italy on digital innovation. Khazna CEO Hassan Alnaqbi stated the agreement represents a defining step in the company's European expansion. He noted that Khazna and Eni are laying the groundwork for infrastructure that meets the growing demands of AI. Eni's Guido Brusco emphasized the project's strategic importance for Italy and Europe. He highlighted both firms' strengths in innovation, sustainability, and rapid delivery. The HoT also sets the foundational roles and governance for a new Joint Venture Company between Khazna and Eni. This structure will accelerate the delivery of the project.
The National
13 hours ago
- The National
Microsoft's president says UAE AI partnership will advance Middle East prosperity
Microsoft's recent AI investments and partnerships in the UAE have the potential to bolster economies throughout the Middle East, the company's president has said. Brad Smith, who also serves as vice chairman of the Redmond, Washington-based technology giant, also said that various US endeavours with the UAE can help to bring technology to other parts of the world. "I believe and hope it will be a beginning that, you know, will advance economic development and prosperity and societal good in the Middle East itself, in places like the UAE, and Saudi Arabia, and Qatar, and the like," he said on Wednesday during an interview with the Centre for Strategic and International Studies (CSIS) in Washington. Over the past decade, the UAE − the Arab world's second largest economy − has sought be a leader in the AI sector as it diversifies its economy away from oil. Microsoft has been helping to support the country's AI aspirations in recent years. The company made a $1.5 billion investment in UAE AI and cloud company G42 in 2024, and later announced that it would open its research-based AI for Good Lab in Abu Dhabi. Mr Smith's comments about Microsoft's projects and programmes in the UAE come several weeks after he told Congress that the US should try to emulate the UAE's AI strategy. He praised Abu Dhabi's Tamm government services AI assistant, which acts as a one-stop-shop for government services including transport, health care, housing and police services. "We need to bring it to America," he told a US Senate commerce, science and transport committee hearing, referring to the need for apps that simplify the process of renewing driver licences, obtaining various forms and other services. During his interview with CSIS, Mr Smith also spoke in greater detail about Microsoft's investment and partnership with UAE's G42. He said the "financial and technological" relationship between the two companies had the potential to pay altruistic dividends around the world. "How do you take AI, which requires electricity, and bring it to countries and to people that don't even have electricity?" he said. "One is a financial and technology partnership, like what we are advancing between the US and the UAE, Microsoft and G42, so that G42 can build out datacenter infrastructure in Africa." During the interview, Mr Smith also addressed the increasingly controversial topic of export controls. "The US has the opportunity to become the world's leading exporter of not just digital technology services, but AI services in the future," he said. Over the last year, Microsoft has been a vocal critic of US export control policies which seek to prevent US AI technology from potentially being used by countries it views as adversarial, like China. Caught in the middle, however, were countries like the UAE, which were disproportionately affected by certain rules that would have limited their ability to obtain the chips required to fulfil their AI goals. In February, Mr Smith said the export rules, created under former president Joe Biden, would cause ally countries to "worry that an insufficient supply of critical American AI technology will restrict their opportunities for economic growth". A recent deal between the UAE and US, called the US-UAE AI Acceleration Partnership, largely alleviated many of those concerns, providing the UAE with a path to obtain the powerful chips. President Donald Trump's administration said that security guarantees within the partnership aiming to prevent diversion of US technology into the wrong hands played a crucial role in making the deal possible. While speaking at CSIS, Mr Smith also briefly touched upon the AI Acceleration Partnership, as well as Microsoft's investment with G42 which preceded it. "Let's do a better job of packaging ourselves," he said, talking about the importance of boasting the benefits of partnerships between companies and countries. "That's in effect what Microsoft and G42 in the US and UAE have started to do, let's continue to move in that direction."
