Perplexity accused of scraping websites even when told not to — here's their response
Cloudflare, in an online blog, published research that showed Perplexity has been crawling and scraping content from websites that explicitly stated they don't want to be scraped.
The research accuses Perplexity of obscuring its identity when trying to scrape web pages, stating that they had received complaints from customers who had both disallowed Perplexity from analysing their files and created rules to specifically block Perplexity from doing this.
Cloudflare performed its own tests to confirm this, creating brand new domains and then querying Perplexity with questions about these specific domains. Perplexity was able to answer queries on these pages, even though Cloudflare had stated it didn't want these websites to be analyzed.
How Perplexity is able to get around these rules is complicated. It appears that Perplexity is changing its bots 'user agent'. In other words, it is pretending to not be a large AI model but just a normal visitor.
Perplexity and lots of other AI tools require large amounts of information to work. They analyse the internet, looking at forums, web pages, and other online sources of information to work.
However, there is more and more backlash to this approach and an expectation for transparency from AI companies on how they gather data. Some of Perplexity's competitors, like Claude and ChatGPT are offering ways to opt out of data gathering, and it is likely we'll see more rules as time goes on.
How Perplexity is able to get around these rules is complicated. It appears that Perplexity is changing its bots 'user agent'. In other words, it is pretending to not be a large AI model but just a normal visitor.
Get instant access to breaking news, the hottest reviews, great deals and helpful tips.
'This activity was observed across tens of thousands of domains and millions of requests per day. We were able to fingerprint this crawler using a combination of machine learning and network signals,' says Cloudflare's post.
Jesse Dwyer, a spokesperson for Perplexity, accused Cloudflare's blog of being a sales pitch for the company in an email to TechCrunch on the subject.
She went on to say that the screenshots in the blog 'show that no content was accessed' and that the bot named in the Cloudflare blog 'isn't even ours'.
Cloudflare is now taking a strong stance on AI crawlers, including Perplexity. The company has claimed that AI is breaking the business model of the internet and wants to help fight back.
While Perplexity has denied this incident, the company has been in hot water before for similar problems, being accused of stealing news sites' content and struggling to define plagiarism.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
an hour ago
- Yahoo
刀仔鋸大樹？Perplexity 出價超自身近兩倍，邀約收購 Google Chrome 瀏覽器
Perplexity AI logo is seen in this illustration taken January 4, 2024. REUTERS/Dado Ruvic/Illustration 人工智能搜尋引擎新星 Perplexity 發出一項震撼科技圈的無預警提案，擬以 345 億美元現金收購瀏覽器 Google Chrome。去年 Google 在美國被判壟斷搜尋市場，並正尋求上訊之際，美國司法部就建議分拆這市佔達 68% 的 Google Chrome 瀏覽器業務，當時 OpenAI 和一眾出席聽證會的科技公司代表，都表示有興趣競投，不過想不到 Perplexity 已經正式發出邀約。 Perplexity 向 TechCrunch 透露部分邀約內容，包括： 保持 Chrome 背後的開源引擎 Chromium 開放及持續投資，承諾斥資 30 億美元支持該開源項目。 承諾不更改現有 Chrome 用戶的預設設定，包括維持 Google 作為預設搜尋引擎，不強制改用自家的 AI 搜尋引擎。 Perplexity 自身已籌資約 15 億美元，目前估值約 180 億美元，此次出價遠超出自身籌資總額及估值。 Perplexity 已經推出了自家瀏覽器 Comet 來擴展其 AI 搜尋業務，但如果能夠收購 Chrome 瀏覽器，更能一舉成為霸主。是說，Perplexity 相當積極佈局科技生態圈，包括曾有意與 TikTok 合併，。 有趣的是，競爭對手 Duck Duck Go 執行長曾在 2025 年 4 月作證，認為 Chrome 價值可能超過 500 億美元；若以此估值看，Perplexity 此次出價雖然已經超過自家估值兩倍，但仍然相當「划算」。 Google 對此收購提案未有回應，並且曾經表示堅持不出售 Chrome，正在法院上訴對其非法壟斷的裁決。市場普遍關注美國司法部是否將於近日對該案提出具體整改方案，屆時可能引發更多企業競價 Chrome。 更多內容： Perplexity offers to buy Chrome for billions more than it's raised Google 若被迫分拆 Chrome，OpenAI：「我買！」 美國聯邦法官再裁定 Google 在線上廣告技術具有壟斷性 緊貼最新科技資訊、網購優惠，追隨 Yahoo Tech 各大社交平台！ 🎉📱 Tech Facebook： 🎉📱 Tech Instagram： 🎉📱 Tech WhatsApp 社群： 🎉📱 Tech WhatsApp 頻道： 🎉📱 Tech Telegram 頻道：
Tom's Guide
an hour ago
- Tom's Guide
I ditched Claude for GPT-5 because of these 5 features — and I know I'll use them every day
For the past few months, Claude has been my go-to chatbot. It used to be ChatGPT, and was for years, but with Anthropic's latest update, its service just felt better than the competition. However, OpenAI's answer in GPT-5 is now here. And it has received a mixed reception. Some loyal ChatGPT fans just can't get on with the update, and OpenAI at first didn't even let you use the old version once you had GPT-5. That has now changed, and OpenAI has stated that they are working hard to iron out any problems with GPT-5. However, I've since gone back to ChatGPT with this new update, and there are a few reasons why. Since GPT-5 came out, it has already gone through some of the major benchmark tests. These are examinations that AI models can be put through, testing their ability on mathematics, coding, writing, emotional intelligence, and more. So far, GPT-5 has managed to come out on top of a lot of these ranking systems. While it has fallen short on a few of them, including SimpleBench, a test comparing the model against human intelligence, for the most part it is now the leading option in the world of AI. One of the main tasks I use chatbots for is writing. Whether it is examining my own writing to check for errors or improve its quality, or helping me come up with inspiration for a given topic, it has quickly become my favorite part of AI. While ChatGPT was never bad at this, I always preferred the style that Claude would generate. It felt more assured and would take on the stylings that I requested in my prompts. However, one of the main improvements that came with GPT-5 was in the model's creativity and writing prowess. OpenAI claims to have made considerable changes to ChatGPT's ability to write creatively and understand more complicated writing prompts. Get instant access to breaking news, the hottest reviews, great deals and helpful tips. In my tests so far, GPT-5 seems more competent in this area, able to write from multiple perspectives in one piece of text, and truly understand more complicated writing styles. The ability to code through chatbots has become a big selling point in recent months. Each model is competing to be the best one at it, and with GPT-5, OpenAI appears to have, at least for me, taken back the crown. It is a very close race with Grok 4, with a split of rankings showing each of the two in the top spot. However, paired with the other features, like its writing abilities, GPT-5 just about takes the win for me. Coding through AI has become a feature I'm using more and more. While I have mainly been using Grok to do this, the GPT-5 update is making me reconsider this, and hopefully, can match the experience I've been having with xAI's tool. Surprisingly, Claude doesn't have the ability to create images. This seems surprising considering how common this feature has become across the different chatbots on the market, but it plays mainly the focus Anthropic wants Claude to have. However, ChatGPT continues to have one of the best AI image generators on the market. Having all of this in one place helps to make ChatGPT a more compelling sell for me. While I don't use image generation as much, I do find it can be really useful for creating graphics alongside reports generated in deep research. This is something GPT-5 should hopefully be able to do well.
Yahoo
2 hours ago
- Yahoo
AI start-up Perplexity makes surprise bid for Google Chrome
Artificial intelligence (AI) start-up Perplexity has made a surprise $34.5bn (£25.6bn) takeover bid for Google's Chrome internet browser. Moving Chrome to an independent operator committed to user safety would benefit the public, Perplexity said in a letter to Sundar Pichai, the boss of Google's owner Alphabet. But one technology industry investor called the offer a "stunt" that is a much lower than Chrome's true value and highlighted that it is not clear whether the platform would is even for sale. The BBC has contacted Google for comment. The firm has not announced any plans to sell Chrome - the world's most popular web browser with an estimated three billion-plus users. Google's dominance of the search engine and online advertising market has come under intense scrutiny, with the technology giant embroiled in years of legal wrangling as part of two antitrust cases. A US federal judge is expected to issue a ruling this month that could see Google being ordered to break up its search business. The company has said it would appeal such a ruling, saying the idea of spinning off Chrome was an "unprecedented proposal" that would harm consumers and security. A spokesman for Perplexity told the BBC that its bid marks an "important commitment to the open web, user choice, and continuity for everyone who has chosen Chrome." As part of the proposed takeover, Perplexity said it would continue to have Google as the default search engine within Chrome, though users could adjust their settings. The firm said it would also maintain and support Chromium, a widely-used open-source platform that supports Chrome and other browsers including Microsoft Edge and Opera. Perplexity did not respond to queries about how the proposed deal would be funded. In July, it had an estimated value of $18bn. Technology industry investor and start-up founder Heath Ahrens called Perplexity's move a "stunt, and nowhere near Chrome's true value, given its unmatched data and reach." "The offer isn't serious, but if someone like Sam Altman or Elon Musk tripled it, they could genuinely secure dominance for their AI," he added. It is also not clear whether Google is considering selling the platform, Tomasz Tunguz from Theory Ventures told the BBC. He also said the offer is a lot lower than the browser is worth "given the value of Chrome is likely significantly higher – maybe ten times more valuable than the bid or more." Perplexity's app is among the rising players in the generative AI race, alongside more well-known platforms like OpenAI's ChatGPT and Google's Gemini. Last month, it launched an AI-powered browser called Comet. The company made headlines earlier this year after offering to buy the American version of TikTok, which faces a deadline in September to be sold by its Chinese owner or be banned in the US. Perplexity has reportedly drawn interest from technology giants including Apple and Facebook-owner Meta.
