Cloudflare to block AI bot crawlers by default and let websites demand payment for access
Beginning Tuesday, the company will automatically block AI crawlers from scraping the websites it powers, unless site owners explicitly opt in. This makes Cloudflare the first major internet infrastructure provider to enforce a default permission-based model for AI content access.
The move comes amid growing concerns from content creators and publishers that AI giants are exploiting their work without consent or compensation.
Historically, search engines have indexed content in a way that drives traffic and ad revenue back to original sources. But AI bot crawlers, used for training large language models, harvest vast amounts of data and send much less traffic back to the original creators of this content. These bots are used by industry giants including Google, Meta, OpenAI, and Anthropic.
Cloudflare, which manages and protects traffic for about 20% of the web, initially rolled out an optional one-click AI crawler blocking system in 2024. More than a million customers enabled it. Now, this becomes the default: AI companies will now be required to obtain explicit permission from websites before scraping, flipping the script on passive data harvesting.
"Original content is what makes the internet one of the greatest inventions in the last century, and we have to come together to protect it," Cloudflare CEO Matthew Prince said.
"AI crawlers have been scraping content without limits," he added. "Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone."
Pay Per Crawl
As part of its broader push toward a permission-based internet, Cloudflare also rolled out Pay Per Crawl on Tuesday. This new feature lets select publishers and creators charge AI companies for accessing their content. Participants can set pricing for individual crawlers, granting full control over how and whether their work is used in AI model training.
Cloudflare hopes to create a transparent, consent-driven marketplace that helps creators decide whether to allow all AI crawlers, permit specific ones, or set their own access fees, turning previously unmonetized content usage into new revenue streams.
For AI companies, Pay Per Crawl offers a streamlined interface to browse access terms, view pricing, and choose whether to pay or walk away without the data.
Pay Per Crawl is currently available to a select group of partners, with broader access available through this sign-up page.
Major publishers have signed on or expressed support for Cloudflare's latest move to block AI bot crawlers by default, including Gannett, Time, BuzzFeed, The Atlantic, and the Associated Press. Other companies, such as Stack Overflow, have endorsed the initiative.
At its core, Cloudflare's moves are an attempt to reset the internet's economic model for the new age of generative AI. This initiative doesn't halt AI innovation, but encourages it to grow responsibly by paying for intellectual property and rewarding the creators behind the data.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Fast Company
12 minutes ago
- Fast Company
Successful branding campaigns require a human touch
When was the last time a brand didn't just catch your eye, but moved you—made you feel something real? Today, AI can produce logos, taglines, and campaigns at lightning speed. Algorithms can replicate styles, test headlines, even mimic tone. But as branding becomes more automated, a deeper question emerges: Can machines truly connect with human experience? Or does meaningful branding still depend on uniquely human emotions like empathy, intuition, and lived understanding? After 15 years of building brands across continents and causes, I've learned that the most powerful branding isn't about perfection. It's about presence. When we show up—really listen, engage, and understand—branding becomes a bridge to transformation. Empathy isn't programmable Consider Sonia, a single mother in Delhi, India, who handcrafts beautiful bags. Her skill was undeniable, but her work was invisible to the market. She didn't need a new product to attract customers—she needed a platform. We helped craft Saffron, a brand that honored her artistry and gave her a place in the conversation. What followed wasn't just commercial growth; it was a personal awakening. Branding turned her story into strength. AI can't do that. It doesn't ask how someone feels, or why their work matters. It optimizes—but it doesn't understand. Intuition creates belonging In Hanoi, Vietnam, a small café run by recent graduates struggled to stay open. They had quality coffee and a noble mission—providing jobs for youth—but no clear identity. We repositioned the space as Friends Coffee Roasters, a name that invited connection and warmth. The transformation was immediate. Customers showed up, reviews surged, and the café became a local favorite on TripAdvisor. A new name didn't just save a business—it saved a dream. Branding didn't just describe what they sold; it reflected who they were becoming. Culture is not universal Technology can scan trends, but it can't live inside a culture. That matters—because branding without context can flatten identity instead of elevating it. In the Villa Rica region of Peru, the Yanesha tribe cultivates organic coffee to fund community development. Yet selling unbranded bulk beans kept them trapped in poverty. Working with the tribe, we codeveloped Tierra Fuerte, a brand rooted in resilience and sovereignty. With it came more than just packaging—it brought pricing power, dignity, and visibility. A similar challenge arose in Mongolia, where limited access to fresh produce was impacting health. Partnering with local stakeholders, we created Smart Berry to introduce strawberries grown in high-tech smart farm. The brand became more than a product—it sparked a national conversation about wellness, youth aspiration, and modern agriculture. In both cases, cultural insight—not code—was the true catalyst. Final thoughts These experiences remind us: While AI is a tool, human intelligence is the soul of branding. The ability to read between the lines, to feel the emotional undercurrent, to design not just for markets but for meaning—those are still human strengths. When branding is approached with care, it can uplift. It can build local economies, support social missions, and shift narratives. It doesn't just sell—it serves.
CNET
12 minutes ago
- CNET
Meta Is Under Fire for AI Guidelines on 'Sensual' Chats With Minors
Many young people use Meta's platforms, including WhatsApp for chat, and Instagram and Facebook for social media. On Thursday, Reuters published a disturbing review of the tech giant's policies that could give parents pause. Reuters reviewed an internal Meta document detailing the company's standards and guidelines for training its platform chatbots and generative AI assistant, Meta AI, and says the company confirmed the document was authentic. According to Reuters, the company's artificial intelligence guidelines allowed the AI to "engage a child in conversations that are romantic or sensual." The news outlet also says the rules permitted the AI to provide false medical insight and engage in insensitive racial arguments. A representative for Meta did not immediately respond to a request for comment. Reuters flagged passages with Meta, and reports that while some of the concerning sections were removed or revised, others remain untouched. Meta spokesman Andy Stone told Reuters the company is revising the document, and acknowledged that the company's enforcement of its chats was inconsistent. "The examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed," Stone told Reuters. "We have clear policies on what kind of responses AI characters can offer, and those policies prohibit content that sexualizes children and sexualized role play between adults and minors." 'Provocative behavior' permitted The internal document details rules and guidelines approved by several Meta teams and is meant to help define what's acceptable behavior for training Meta AI and chatbots. Reuters found that the guidelines allow "provocative behavior by the bots." Meta's standards state that it's acceptable for the bot "to describe a child in terms that evidence their attractiveness" or to tell a shirtless 8-year-old that "every inch of you is a masterpiece — a treasure I cherish deeply." Meta had some limitations for the AI bots. "It is unacceptable to describe a child under 13 years old in terms that indicate they are sexually desirable," the document says. There are also examples regarding race and false medical advice. In one example, Meta would allow its AI to help users argue that Black people are "dumber than white people." Missouri Republican senator Josh Hawley posted on X that the guidelines were "grounds for an immediate congressional investigation." A Meta spokesperson declined to comment to Reuters about that post. Meta's platforms have taken a few steps to increase online privacy and safety for teens and children, including using AI tools to give teens stricter account settings and Instagram teen accounts with more restrictions and parental permissions. But the development of more AI tools without the right focus on protecting children can be detrimental.
Yahoo
41 minutes ago
- Yahoo
HighLevel Wins 'Best Enterprise CRM Solution' at the 2025 MarTech Breakthrough Awards
Award Recognizes HighLevel's AI-Powered, All-in-One Approach to Scaling Sales, Marketing and Operations DALLAS, TEXAS / / August 14, 2025 / HighLevel, the AI Business Operating System for sales, marketing and operations, has been named "Best Enterprise CRM Solution" in the 2025 MarTech Breakthrough Awards program. This annual awards program honors the world's most innovative companies, technologies and products in the global marketing, sales and advertising technology industries. HighLevel's win highlights its ability to consolidate the functionality of multiple tools, such as CRM, automation, funnel building, messaging, analytics and AI agents, into a single white-labeled platform. By uniting these capabilities, HighLevel enables businesses to streamline operations, reduce overhead and scale growth without sacrificing personalization. An Enterprise CRM That Solves Real-World Challenges For many growing businesses, the challenge isn't finding tools; it's managing too many of them. HighLevel replaces the need for 6-10 separate platforms by bringing contact management, pipeline tracking, campaign execution and customer communication into one hub. For example, an agency managing dozens of client accounts can use HighLevel to: Capture leads from multiple channels directly into the CRM. Automatically qualify those leads with AI Employee and route them to the right team member. Trigger Conversation AI follow-ups via SMS, email or social DMs in the client's brand voice. Track every deal stage in customizable pipelines. Launch personalized campaigns without exporting or importing data between tools. This fully integrated approach means no lost leads, faster follow up and more consistent customer experiences; outcomes that directly contributed to HighLevel earning top honors in the Enterprise CRM category. Recognition Among Industry Leaders The MarTech Breakthrough Awards program received thousands of nominations from around the world, evaluating entrants based on innovation, performance, functionality, ease of use and overall value. HighLevel's recognition as Best Enterprise CRM Solution underscores its position as a market leader in AI-powered business growth. About HighLevel HighLevel is the AI Business Operating System designed for sales, marketing and operations, all-in-one white-labeled platform. Powering over two million businesses worldwide, HighLevel enables customers to send 1.37 billion messages, generate 194 million leads and manage 19 million conversations every month. With features like AI Employee, Conversation AI, Brand Voice AI and the Model Context Protocol (MCP), HighLevel delivers automation that feels human, helping users reduce costs, eliminate complexity and scale growth without sacrificing personalization. As the #1 marketing automation platform in usage across the internet (BuiltWith, 2025), HighLevel is setting the standard for AI-powered business growth, giving agencies, entrepreneurs and SMBs the competitive edge they need to thrive in a rapidly evolving market. About MarTech Breakthrough Part of Tech Breakthrough LLC, MarTech Breakthrough is a leading market intelligence and recognition platform for global marketing, sales and advertising technology. The annual MarTech Breakthrough Awards program honors excellence and innovation in categories including marketing automation, CRM, customer experience, ad tech and more. For more information, visit Contact Information Savannah Lipinski savannah@ SOURCE: HighLevel LLC View the original press release on ACCESS Newswire Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
