Millions of websites to get 'game-changing' AI bot blocker
Cloudflare's tech targets AI firm bots - also known as crawlers - programmes that explore the web, indexing and collecting data as they go. They are important to the way AI firms build, train and operate their systems.So far, Cloudflare says its tech is active on a million websites.Roger Lynch, chief executive of Condé Nast, whose print titles include GQ, Vogue, and The New Yorker, said the move was "a game-changer" for publishers."This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable", he wrote in a statement.However, other experts say stronger legal protections will still be needed.
'Surviving the age of AI'
Initially the system will apply by default to new users of Cloudflare services, plus sites that participated in an earlier effort to block crawlers.Many publishers accuse AI firms of using their content without permission.Recently the BBC threatened to take legal action against US based AI firm Perplexity, demanding it immediately stopped using BBC content, and paid compensation for material already used.However publishers are generally happy to allow crawlers from search engines, like Google, to access their sites, so that the search companies can in return can direct people to their content. Perplexity accused the BBC of seeking to preserve "Google's monopoly". But Cloudflare argues AI breaks the unwritten agreement between publishers and crawlers. AI crawlers, it argues, collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue. "If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone," wrote the firm's chief executive Matthew Prince. To that end the company is developing a "Pay Per Crawl" system, which would give content creators the option to request payment from AI companies for utilising their original content.
Battle the bots
According to Cloudflare there has been an explosion of AI bot activity. "AI Crawlers generate more than 50 billion requests to the Cloudflare network every day", the company wrote in March.And there is growing concern that some AI crawlers are disregarding existing protocols for excluding bots.In an effort to counter the worst offenders Cloudflare previously developed a system where the worst miscreants would be sent to a "Labyrinth" of web pages filled with AI generated junk.The new system attempts to use technology to protect the content of websites and to charge AI firms to access it.In the UK there is an intense legislative battle between government, creators and the AI firms over the extent to which the creative industries should be protected from AI firms using their works to train systems without permission or payment.And, on both sides of the Atlantic, content creators, licensors and owners have gone to court in an effort to prevent what they see as AI firms encroachment on creative rights.Ed Newton-Rex, the founder of Fairly Trained which certifies that AI companies have trained their systems on properly licensed data, said it was a welcome development - but there was "only so much" one company could do "This is really only a sticking plaster when what's required is major surgery," he told the BBC."It will only offer protection for people on websites they control - it's like having body armour that stops working when you leave your house," he added."The only real way to protect people's content from theft by AI companies is through the law."
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
The Guardian
an hour ago
- The Guardian
Aukus deal's importance ‘well understood' by Trump administration, Penny Wong says – but no iron-clad guarantee given
The strategic importance of the Aukus agreement is 'well understood' by the Trump administration, Penny Wong says, despite a snap Pentagon review casting doubt on the future of the pact. In Washington DC for talks with Quad nations foreign ministers on Wednesday, Australia's foreign affairs minister said the US secretary of state, Marco Rubio, had heard Australia's position on the need for the agreement to proceed, though she did not receive any iron-clad assurances from her counterpart. The Aukus deal outlines the sharing of nuclear-powered submarine technology between Australia, the US and the UK at a cost of about A$350bn. The federal government has already made payments worth A$800m. Despite the review being led by an Aukus sceptic and US undersecretary of defense, Elbridge Colby, Wong said she remained confident the partnership would 'last decades'. Sign up for Guardian Australia's breaking news email She said Australia and the US understood 'the importance to both our countries and to the United Kingdom'. 'We understand a new administration is going to engage in a review,' Wong said. 'That is not surprising. The United Kingdom did so. 'This is a multi-decade partnership. It will take governments and administrations of both political persuasions over many years to deliver. We will continue to work with the United States and the United Kingdom on delivering it.' The talks come as Donald Trump and the US defense secretary, Pete Hegseth, push Labor to lift Australia's defence spending to 3.5% of GDP. The federal government is resisting that pressure. Australia is on track to lift defence spending from about $53bn a year, or about 2% of GDP – to an estimated $100bn, or 2.4% of GDP, by 2033-34. The Quad talks included Japan's foreign minister, Iwaya Takeshi, and India's external affairs minister, Subrahmanyam Jaishankar. Sign up to Breaking News Australia Get the most important news as it breaks after newsletter promotion Wong said Rubio had expressed regret that Trump had not met with Anthony Albanese at the G7 summit last month. Trump left the talks in Canada a day early to deal with the war between Israel and Iran. 'I explained to him we perfectly understood why the president had to leave, given the circumstances, and we agreed that we will reschedule this meeting,' she said. 'We both look forward to the leaders meeting. 'I think that the Trump administration and President Trump has made very clear to the world that he envisages a different role for America in the world.' One possible opportunity for a meeting between Trump and the prime minister at the White House could be after the UN general assembly in New York in September. India will host the next Quad leaders' summit later this year. Australia could benefit from demand for critical minerals from Quad countries including Japan and India, with exports forming part of efforts to secure supply chains as a balance against China's domination of the market. Rubio said the US wanted to diversify global supplies and processing. 'It's critical for all technologies and for all industries across the board,' he said. 'Having a diverse and reliable global supply chain of these is just one example of many that we can focus on and build upon and achieve some real progress on.' Wong called progress on critical minerals part of efforts to secure 'concrete outcomes' for a stable and peaceful Indo-Pacific region. 'This is a supply set of supply chains we need to assure. Australia has, I think, 36 of the 50 critical minerals. We certainly have a capacity to do more for those, to secure those supply chains.'
The Guardian
an hour ago
- The Guardian
US halts weapons shipments to Ukraine over fears stockpiles are too low
The US is halting some shipments of weapons to Ukraine amid concerns that its own stockpiles have declined too much, officials said Tuesday, a setback for the country as it tries to fend off escalating attacks from Russia. Certain munitions were previously promised to Ukraine under the Biden administration to aid its defences during the more than three-year-old war. The pause reflects a new set of priorities under President Donald Trump and came after defence department officials scrutinised US stockpiles and raised concerns. 'This decision was made to put America's interests first following a review of our nation's military support and assistance to other countries across the globe,' White House spokesperson Anna Kelly said in a statement. 'The strength of the United States Armed Forces remains unquestioned – just ask Iran.' That was a reference to Trump recently ordering US missile strikes against nuclear sites in Iran. The Pentagon review determined that stocks were too low on some weapons previously pledged, so pending shipments of some items won't be sent, according to a US official who spoke on condition of anonymity to provide information that has not yet been made public. The defence department did not provide details on what specific weapons were being held back. 'America's military has never been more ready and more capable,' spokesperson Sean Parnell said, adding that the major tax cut and spending package moving through Congress 'ensures that our weapons and defence systems are modernised to protect against 21st century threats for generations to come.' The halt of some weapons from the US is a blow to Ukraine as Russia has recently launched some of its biggest aerial attacks of the war, in an escalating bombing campaign that has further dashed hopes for a breakthrough in peace efforts championed by Trump. Talks between the sides have ground to a halt. The US stoppage was first reported by Politico. To date, the US has provided Ukraine more than $66bn worth of weapons and military assistance since Russia invaded its neighbour in February 2022. Over the course of the war, the US has routinely pressed for allies to provide air defence systems to Ukraine. But many are reluctant to give up the hi-tech systems, particularly countries in eastern Europe that also feel threatened by Russia. Trump met with Ukrainian leader Volodymyr Zelenskyy on the sidelines of the Nato summit last week and had left open the possibility of sending Kyiv more US-made Patriot air defence missile systems, acknowledging they would help the Ukrainian cause. 'They do want to have the antimissile missiles, OK, as they call them, the Patriots,' Trump said then. 'And we're going to see if we can make some available. We need them, too. We're supplying them to Israel, and they're very effective, 100% effective. Hard to believe how effective. They do want that more than any other thing.' Those comments reflect a change of thinking about providing weapons to Ukraine across the administration in recent months. In testimony before lawmakers in June, defence secretary Pete Hegseth said he had moved quickly to quash wasteful programmes and redirect funding to Trump's top objectives. Hegseth said a negotiated peace between Russia and Ukraine, which has been promoted for months by Trump, makes America look strong, even though Moscow is the aggressor in the conflict. He also said the defence budget includes hard choices and 'reflects the reality that Europe needs to step up more for the defence of its own continent. And President Trump deserves the credit for that.' The defense secretary told lawmakers last month that some US security spending for Ukraine was still in the pipeline, without providing details. But he said such assistance – which has been robust for the past two years – would be reduced.
Sky News
an hour ago
- Sky News
The PM faced down his party on welfare and lost. I suspect things may only get worse
So much for an end to chaos and sticking plaster politics. Yesterday, Sir Keir Starmer abandoned his flagship welfare reforms at the eleventh hour - hectic scenes in the House of Commons that left onlookers aghast. Facing possible defeat on his welfare bill, the PM folded in a last-minute climbdown to save his skin. 0:23 The decision was so rushed that some government insiders didn't even know it was coming - as the deputy PM, deployed as a negotiator, scrambled to save the bill or how much it would cost. "Too early to answer, it's moved at a really fast pace," said one. The changes were enough to whittle back the rebellion to 49 MPs as the prime minister prevailed, but this was a pyrrhic victory. Sir Keir lost the argument with his own backbenchers over his flagship welfare reforms, as they roundly rejected his proposed cuts to disability benefits for existing claimants or future ones, without a proper review of the entire personal independence payment (PIP) system first. PM wins key welfare vote - follow latest 4:31 That in turn has blown a hole in the public finances, as billions of planned welfare savings are shelved. Chancellor Rachel Reeves now faces the prospect of having to find £5bn. As for the politics, the prime minister has - to use a war analogy - spilled an awful lot of blood for little reward. He has faced down his MPs and he has lost. 4:38 They will be emboldened from this and - as some of those close to him admit - will find it even harder to govern. After the vote, in central lobby, MPs were already saying that the government should regard this as a reset moment for relations between No 10 and the party. The prime minister always said during the election that he would put country first and party second - and yet, less than a year into office, he finds himself pinned back by his party and blocked from making what he sees are necessary reforms. I suspect it will only get worse. When I asked two of the rebel MPs how they expected the government to cover off the losses in welfare savings, Rachael Maskell, a leading rebel, suggested the government introduce welfare taxes. Meanwhile, Work and Pensions Select Committee chair Debbie Abrahams told me "fiscal rules are not natural laws" - suggesting the chancellor could perhaps borrow more to fund public spending. 0:45 These of course are both things that Ms Reeves has ruled out. But the lesson MPs will take from this climbdown is that - if they push hard in enough and in big enough numbers - the government will give ground. The fallout for now is that any serious cuts to welfare - something the PM says is absolutely necessary - are stalled for the time being, with the Stephen Timms review into PIP not reporting back until November 2026. 1:10 Had the government done this differently and reviewed the system before trying to impose the cuts - a process only done ahead of the Spring Statement in order to help the chancellor fix her fiscal black hole - they may have had more success. Those close to the PM say he wants to deliver on the mandate the country gave him in last year's election, and point out that Sir Keir Starmer is often underestimated - first as party leader and now as prime minister. But on this occasion, he underestimated his own MPs. His job was already difficult enough - and after this it will be even harder still. If he can't govern his party, he can't deliver change he promised.
