
AI Just Hit A Paywall As The Web Reacts To Cloudflare's Flip
As someone who has spent years building partnerships between tech innovators and digital creators, I've seen how difficult it can be to balance visibility and value. Every week, I meet with founders and business leaders trying to figure out how to stand out, monetize content, and keep control of their digital assets. They're proud of what they've built but increasingly worried that AI systems are consuming their work without permission, credit, or compensation.
That's why Cloudflare's latest announcement hit like a thunderclap. And I wanted to wait to see the responses from companies and creators to really tell this story.
Cloudflare, one of the internet's most important infrastructure companies, now blocks AI crawlers by default for all new customers.
This flips the longstanding model, where crawlers were allowed unless actively blocked, into something more deliberate: AI must now ask to enter.
And not just ask. Pay.
Cloudflare wants AI Companies to pay for the data that they use.
Alongside that change, Cloudflare has launched Pay‑Per‑Crawl, a new marketplace that allows website owners to charge AI companies per page crawled. If you're running a blog, a digital magazine, a startup product page, or even a knowledge base, you now have the option to set a price for access. AI bots must identify themselves, send payment, and only then can they index your content.
This isn't a routine product update. It's a signal that the free ride for AI training data is ending and a new economic framework is beginning.
AI Models and Their Training
The core issue behind this shift is how AI models are trained. Large language models like OpenAI's GPT or Anthropic's Claude rely on huge amounts of data from the open web. They scrape everything, including articles, FAQs, social posts, documentation, even Reddit threads, to get smarter. But while they benefit, the content creators see none of that upside.
Unlike traditional search engines that drive traffic back to the sites they crawl, generative AI tends to provide full answers directly to users, cutting creators out of the loop.
According to Cloudflare, the data is telling: OpenAI's crawl-to-referral ratio is around 1,700 to 1. Anthropic's is 73,000 to 1. Compare that to Google, which averages about 14 crawls per referral, and the imbalance becomes clear.
In other words, AI isn't just learning from your content but it's monetizing it without ever sending users back your way.
Rebalancing the AI Equation
Cloudflare's announcement aims to rebalance this equation. From now on, when someone signs up for a new website using Cloudflare's services, AI crawlers are automatically blocked unless explicitly permitted. For existing customers, this is available as an opt-in.
More importantly, Cloudflare now enables site owners to monetize their data through Pay‑Per‑Crawl. AI bots must:
Only then will the content be served.
We have to rebalance the AI equation.
This marks a turning point. Instead of AI companies silently harvesting the web, they must now enter into economic relationships with content owners. The model is structured like a digital toll road and this road leads to your ideas, your writing, and your value.
Several major publishers are already onboard. According to Neiman Lab, Gannett, Condé Nast, The Atlantic, BuzzFeed, Time, and others have joined the system to protect and monetize their work.
Cloudflare Isn't The Only One Trying To Protect Creators From AI
This isn't happening in a vacuum. A broader wave of startups and platforms are emerging to support a consent-based data ecosystem.
CrowdGenAI is focused on assembling ethically sourced, human-labeled data that AI developers can license with confidence. It's designed for the next generation of AI training where the value of quality and consent outweighs quantity. (Note: I am on the advisory board of CrowdGenAI).
Real.Photos is a mobile camera app that verifies your photos are real, not AI. The app also verifies where the photo was taken and when. The photo, along with its metadata are hashed so it can't be altered. Each photo is stored on the Base blockchain as an NFT and the photo can be looked up and viewed on a global, public database. Photographers make money by selling rights to their photos. (Note: the founder of Real.Photos is on the board of Unstoppable - my employer)
Spawning.ai gives artists and creators control over their inclusion in datasets. Their tools let you mark your work as 'do not train,' with the goal of building a system where creators decide whether or not they're part of AI's learning process.
Tonic.ai helps companies generate synthetic data for safe, customizable model training, bypassing the need to scrape the web altogether.
DataDistil is building a monetized, traceable content layer where AI agents can pay for premium insights, with full provenance and accountability.
Each of these players is pushing the same idea: your data has value, and you deserve a choice in how it's used.
What Are the Pros to Cloudflare's AI Approach?
There are real benefits to Cloudflare's new system.
First, it gives control back to creators. The default is 'no,' and that alone changes the power dynamic. You no longer have to know how to write a robots.txt file or hunt for obscure bot names.
Cloudflare handles it.
Second, it introduces a long-awaited monetization channel. Instead of watching your content get scraped for free, you can now set terms and prices.
Third, it promotes transparency. Site owners can see who's crawling, how often, and for what purpose. This turns a shadowy process into a visible, accountable one.
Finally, it incentivizes AI developers to treat data respectfully. If access costs money, AI systems may start prioritizing quality, licensing, and consent.
And There Are Some Limitations To The AI Approach
But there are limitations.
Today, all content is priced equally. That means a one-sentence landing page costs the same to crawl as an investigative feature or technical white paper. A more sophisticated pricing model will be needed to reflect actual value.
Enforcement could also be tricky.
Not all AI companies will follow the rules. Some may spoof bots or route through proxy servers. Without broader adoption or legal backing, the system will still face leakage.
There's also a market risk. Cloudflare's approach assumes a future where AI agents have a budget, where they'll pay to access the best data and deliver premium answers. But in reality, free often wins. Unless users are willing to pay for higher-quality responses, AI companies may simply revert to scraping from sources that remain open.
And then there's the visibility problem. If you block AI bots from your site, your content may not appear in agent-generated summaries or answers. You're protecting your rights—but possibly disappearing from the next frontier of discovery.
I was chatting with Daniel Nestle, Founder of Inquisitive Communications, who told me 'Brands and creators will need to understand that charging bots for content will be the same as blocking the bots: their content will disappear from GEO results and, more importantly, from model training, forfeiting the game now and into the future.'
Daniel Nestle, Founder of Inquisitive Communications, on the risks of blocking AI scraping.
The AI Fork In The Road
What Cloudflare has done is more than just configure a setting. They've triggered a deeper conversation about ownership, consent, and the economics of information. The default mode of the internet with free access, free usage, no questions asked, is being challenged.
This is a fork in the road.
One path leads to a web where AI systems must build partnerships with creators. Take the partnership of Perplexity with Coinbase on crypto data. The other continues toward unchecked scraping, where the internet becomes an unpaid training ground for increasingly powerful models.
Between those extremes lies the gray space we're now entering: a space where some will block, some will charge, and some will opt in for visibility. What matters is that we now have the tools and the leverage to make that decision.
For creators, technologists, and companies alike, that changes everything.
Did you enjoy this story about AI And Cloudflare? Don't miss my next one: Use the blue follow button at the top of the article near my byline to follow more of my work.
Hashtags

Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
25 minutes ago
- Yahoo
Lilly confirms date and conference call for second-quarter 2025 financial results announcement
INDIANAPOLIS, July 24, 2025 /PRNewswire/ -- Eli Lilly and Company (NYSE: LLY) will announce its second-quarter 2025 financial results on August 7, 2025. Lilly will also conduct a conference call that day with the investment community and media to further detail the company's financial performance. The conference call will begin at 8:30 a.m. Eastern time. Investors, media and the general public can access a live webcast of the conference call through a link that will be posted on Lilly's website at A replay will also be available on the website following the conference call. About LillyLilly is a medicine company turning science into healing to make life better for people around the world. We've been pioneering life-changing discoveries for nearly 150 years, and today our medicines help tens of millions of people across the globe. Harnessing the power of biotechnology, chemistry and genetic medicine, our scientists are urgently advancing new discoveries to solve some of the world's most significant health challenges: redefining diabetes care; treating obesity and curtailing its most devastating long-term effects; advancing the fight against Alzheimer's disease; providing solutions to some of the most debilitating immune system disorders; and transforming the most difficult-to-treat cancers into manageable diseases. With each step toward a healthier world, we're motivated by one thing: making life better for millions more people. That includes delivering innovative clinical trials that reflect the diversity of our world and working to ensure our medicines are accessible and affordable. To learn more, visit and or follow us on Facebook, Instagram, and LinkedIn. F-LLY Trademarks and Trade NamesAll trademarks or trade names referred to in this press release are the property of the company, or, to the extent trademarks or trade names belonging to other companies are references in this press release, the property of their respective owners. Solely for convenience, the trademarks and trade names in this press release are referred to without the ® and ™ symbols, but such references should not be construed as any indicator that the company or, to the extent applicable, their respective owners will not assert, to the fullest extent under applicable law, the company's or their rights thereto. We do not intend the use or display of other companies' trademarks and trade names to imply a relationship with, or endorsement or sponsorship of us by, any other companies. Refer to: Ashley Hennessey; gentry_ashley_jo@ 317-416-4363 (Media)Michael Czapar; czapar_michael_c@ 317-617-0983 (Investors) View original content to download multimedia: SOURCE Eli Lilly and Company Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Yahoo
25 minutes ago
- Yahoo
Analyst Says Advanced Micro Devices (AMD) Can Get $25 Billion in GPU Sales With Just 5% Market Share
Advanced Micro Devices Inc (NASDAQ:AMD) is one of the 10 Stocks to Buy and Sell in 2025: Top Analyst Calls. Ben Reitzes, Melius Research managing director, recently said in a program on CNBC that Nvidia getting the permission to start selling AI chips in China is also 'great' for Advanced Micro Devices Inc (NASDAQ:AMD). He believes AMD could benefit if it manages to get even 5% of the total market share amid a rise in inference. 'Our thesis on Advanced Micro Devices Inc (NASDAQ:AMD) is I mean how could they—Lisa Su is real. She's great. She's very compelling. Nothing against Jensen, but how do they not get like 5% share of the market? Not even 10. If they get 5% of the market in 2028, that's 25 billion in GPU sales. That's like 950, 10 bucks in earnings. So, our bet on Advanced Micro Devices Inc (NASDAQ:AMD) is they have a good product. It's really good for inferencing and people are going to give them a look. So, we're hearing, you know, like Meta's a little more interested in some of the new stuff. X, XAI, OpenAI really interested. So our thesis there now, China for them is it could be like add 20% plus to their AI revenue, but they don't have—they just have one, the old chip they were selling. We need to hear about more chips to get more confident, but it's great for Advanced Micro Devices Inc (NASDAQ:AMD). Don't get me wrong. It's great for AMD.' Photo by Vishnu Mohanan on Unsplash Advanced Micro Devices (NASDAQ:AMD) bulls believe the market should stop comparing the company's chips with Nvidia and focus on its data-center growth and its competitive edge over other players like Intel. Advanced Micro Devices (NASDAQ:AMD)'s strong growth in the data center segment is indeed impressive, driven by Instinct GPU shipments and strong sales of EPYC CPUs. Advanced Micro Devices (NASDAQ:AMD) will continue to benefit from organic growth catalysts in this segment despite the competition from Nvidia. According to Goldman Sachs Research, global data center demand could surge by 160% by 2030. In the U.S., data centers are projected to use 8% of total power by 2030, up from 3% in 2022. McKinsey estimates that adding the required U.S. capacity will need over $500 billion in infrastructure investment by the decade's end. Longriver Partners Fund stated the following regarding Advanced Micro Devices, Inc. (NASDAQ:AMD) in its second quarter 2025 investor letter:
Yahoo
25 minutes ago
- Yahoo
Where Will Robinhood Markets Stock Be in 5 Years?
Key Points Robinhood's assets under management (AUM) have grown a stunning 70% in the last 12 months. Management is focused on innovation, which could lead to new revenue streams for the company. 10 stocks we like better than Robinhood Markets › Five years from now, I believe Robinhood Markets (NASDAQ: HOOD) stock will be higher than it is today. Granted, that's a vague prediction, but it's also one I feel pretty confident in. After all, while trying to pinpoint future stock prices is often a useless exercise, identifying long-term trends is far easier. So, what are these long-term trends, and why do they give me such confidence that Robinhood stock will rise over the next five years? Let's dig in and explore them one by one. Robinhood's growth engine First off, to get a grasp of where Robinhood stock might be five years from now, it's important to understand its current state of affairs. The company operates an online brokerage offering customers the ability to trade stocks, options, futures, crypto, and other financial instruments. However, it does not generate most of its revenue through traditional commissions. Instead, the company creates revenue in two main ways: First, it routes customer orders to various market makers, which pay Robinhood. This process is called payment-for-order-flow (PFOF). Second, the company generates net-interest revenue from the cash held in its customers' brokerage accounts. Currently, Robinhood has over $221 billion in assets under management (AUM). Growth of its AUM will drive higher PFOF revenue and net-interest revenue. What's more, Robinhood's AUM is growing at an incredible rate. For example, over the last year, the company's total AUM has grown from $130 billion to $221 billion, an increase of 70%. Robinhood's management Thanks to its business model and its surging popularity, Robinhood is growing rapidly. But, in my opinion, it's the company's management that will propel its stock higher over the next five years. Robinhood is led by co-founder and CEO Vlad Tenev, a 38-year-old Bulgarian-American with a penchant for thinking big. He has announced several innovative endeavors at Robinhood: Artificial intelligence (AI): Integrating AI into investing with the launch of Cortex, Robinhood's AI system. Tokenization: This could provide access for those seeking to monetize (and trade) everything from art to mortgages to private equity shares. Prediction markets: Robinhood has already launched some prediction contracts, allowing users to place bets on real-world outcomes, ranging from politics to economic data. In short, Tanev isn't satisfied with simply running another online brokerage. He wants to push the boundaries of what is possible and expand into other areas adjacent to the traditional brokerage segment. Where will Robinhood stock be in five years? While no one can say for sure where Robinhood stock will be in 2030, I do think the stock will rise from its current level. As of this writing, Robinhood has a market capitalization of around $100 billion. That makes it around the 100th-largest company in America -- roughly the same size as Intel. By 2030, I think RobinHood could move up in that ranking, perhaps challenging one of its rivals, Charles Schwab, which has a market cap of around $175 billion, making it about the 50th-largest American company. In any event, I think Robinhood's robust growth and visionary leadership make it a stock to consider right now. Should you invest $1,000 in Robinhood Markets right now? Before you buy stock in Robinhood Markets, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the for investors to buy now… and Robinhood Markets wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $634,627!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $1,046,799!* Now, it's worth noting Stock Advisor's total average return is 1,037% — a market-crushing outperformance compared to 182% for the S&P 500. Don't miss out on the latest top 10 list, available when you join Stock Advisor. See the 10 stocks » *Stock Advisor returns as of July 21, 2025 Charles Schwab is an advertising partner of Motley Fool Money. Jake Lerch has the following options: long January 2026 $30 calls on Robinhood Markets. The Motley Fool has positions in and recommends Intel. The Motley Fool recommends Charles Schwab and recommends the following options: short August 2025 $24 calls on Intel and short September 2025 $92.50 calls on Charles Schwab. The Motley Fool has a disclosure policy. Where Will Robinhood Markets Stock Be in 5 Years? was originally published by The Motley Fool Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data