THE ECONOMIST: AI is killing the web. Can anything save it?
They told Mr Prince, whose firm, Cloudflare, provides security infrastructure to about a fifth of the web, that their businesses faced a grave new online threat.
'I said, 'What, is it the North Koreans?',' he recalls. 'And they said, 'No. It's AI'.'
Those executives had spotted the early signs of a trend that has since become clear: artificial intelligence is transforming the way that people navigate the web.
As users pose their queries to chatbots rather than conventional search engines, they are given answers, rather than links to follow.
The result is that 'content' publishers, from news providers and online forums to reference sites such as Wikipedia, are seeing alarming drops in their traffic.
As AI changes how people browse, it is altering the economic bargain at the heart of the internet.
Human traffic has long been monetised using online advertising; now that traffic is drying up.
Content producers are urgently trying to find new ways to make AI companies pay them for information.
If they cannot, the open web may evolve into something very different.
Since the launch of ChatGPT in late 2022, people have embraced a new way to seek information online.
OpenAI, maker of ChatGPT, says that around 800 million people use the chatbot.
It is the most popular download on the iPhone app store.
Apple said that conventional searches in its Safari web browser had fallen for the first time in April, as people posed their questions to AI instead.
OpenAI is soon expected to launch a browser of its own. Its rise is so dramatic that a Hollywood adaptation is in the works.
As OpenAI and other upstarts have soared, Google, which has about 90 per cent of the conventional search market in America, has added AI features to its own search engine in a bid to keep up.
Last year it began preceding some search results with AI-generated 'overviews', which have since become ubiquitous.
In May it launched 'AI mode', a chatbot-like version of its search engine. The company promises that, with AI, users can 'let Google do the Googling for you'.
Yet as Google does the Googling, humans no longer visit the websites from which the information is gleaned.
Similarweb, which measures traffic to more than 100m web domains, estimates that worldwide search traffic (by humans) fell by about 15 per cent in the year to June.
Although some categories, such as hobbyists' sites, are doing fine, others have been hit hard.
Many of the most affected are just the kind that might have commonly answered search queries.
Science and education sites have lost 10 per cent of their visitors. Reference sites have lost 15 per cent. Health sites have lost 31 per cent.
For companies that sell advertising or subscriptions, lost visitors means lost revenue.
'We had a very positive relationship with Google for a long time . . . They broke the deal,' says Neil Vogel, head of Dotdash Meredith, which owns titles such as People and Food & Wine. T
hree years ago its sites got more than 60 per cent of their traffic from Google. Now the figure is in the mid-30s.
'They are stealing our content to compete with us,' says Mr Vogel.
Google has insisted that its use of others' content is fair. But since it launched its AI overviews, the share of news-related searches resulting in no onward clicks has risen from 56 per cent to 69 per cent, estimates Similarweb. In other words, seven in ten people get their answer without visiting the page that supplied it.
'The nature of the internet has completely changed,' says Prashanth Chandrasekar, chief executive of Stack Overflow, best known as an online forum for coders.
'AI is basically choking off traffic to most content sites,' he says.
With fewer visitors, Stack Overflow is seeing fewer questions posted on its message boards.
Wikipedia, also powered by enthusiasts, warns that AI-generated summaries without attribution 'block pathways for people to access . . . and contribute to' the site.
To keep the traffic and the money coming, many big content producers have negotiated licensing deals with AI companies, backed up by legal threats: what Robert Thomson, chief executive of News Corp, has dubbed 'wooing and suing'.
His company, which owns the Wall Street Journal and the New York Post, among other titles, has struck a deal with OpenAI. Two of its subsidiaries are suing Perplexity, another AI answer engine.
The New York Times has done a deal with Amazon while suing OpenAI. Plenty of other transactions and lawsuits are going on. (The Economist's parent company has not taken a public position on whether it will licence our work.)
Yet this approach has limits. For one thing, judges so far seem minded to side with AI companies: last month two separate copyright cases in California went in favour of their defendants, Meta and Anthropic, both of which argued that training their models on others' content amounted to fair use.
President Donald Trump seems to accept Silicon Valley's argument that it must be allowed to get on with developing the technology of the future before China can.
He has appointed tech boosters as advisers on AI, and sacked the head of the US Copyright Office soon after she argued that training AI on copyrighted material was not always legal.
AI companies are more willing to pay for continuing access to information than training data. But the deals done so far are hardly stellar.
Reddit, an online forum, has licensed its user-generated content to Google for a reported $US60m ($AU91.5m) a year. Yet its market value fell by more than half — over $US20 billion — after it reported slower user-growth than expected in February, owing to wobbles in search traffic. (Growth has since picked up and Reddit's share price has recovered some lost ground.)
The bigger problem however, is that most of the internet's hundreds of millions of domains are too small to either woo or sue the tech giants. Their content may be collectively essential to AI firms, but each site is individually dispensable.
Even if they could join forces to bargain collectively, antitrust law would forbid it. They could block AI crawlers, and some do. But that means no search visibility at all.
Software providers may be able to help. All of Cloudflare's new customers will now be asked if they want to allow AI companies' bots to scrape their site, and for what purpose.
Cloudflare's scale gives it a better chance than most of enabling something like a collective response by content sites that want to force AI firms to cough up. It is testing a pay-as-you-crawl system that would let sites charge bots an entry fee.
'We have to set the rules of the road,' says Mr Prince, who says his preferred outcome is 'a world where humans get content for free, and bots pay a tonne for it'.
An alternative is offered by Tollbit, which bills itself as a paywall for bots. It allows content sites to charge AI crawlers varying rates: for instance, a magazine could charge more for new stories than old ones.
In the first quarter of this year Tollbit processed 15m micro-transactions of this sort, for 2000 content producers including the Associated Press and Newsweek.
Toshit Panigrahi, its chief executive, points out that whereas traditional search engines incentivise samey content — 'What time does the Super Bowl start?', for example — charging for access incentivises uniqueness. One of Tollbit's highest per-crawl rates is charged by a local newspaper.
Another model is being put forward by ProRata, a startup led by Bill Gross, a pioneer in the 1990s of the pay-as-you-click online ads that have powered much of the web ever since.
He proposes that money from ads placed alongside AI-generated answers should be redistributed to sites in proportion to how much their content contributed to the answer.
ProRata has its own answer engine, Gist.ai, which shares ad revenue with its 500-plus partners, which includethe Financial Times and the Atlantic.
It is currently more of an exemplar than a serious threat to Google: Mr Gross says his main aim is to 'show a fair business model that other people eventually copy'.
Meanwhile, content producers are rethinking their business models.
'The future of the internet is not all about traffic,' says Mr Chandrasekar, who has built up Stack Overflow's private, enterprise-oriented subscription product, Stack Internal.
News publishers are planning for 'Google zero', deploying newsletters and apps to reach customers who no longer come to them via search, and moving their content behind paywalls or to live events.
Audio and video are proving legally and technically harder for AI engines to summarise than text. The site to which answer engines refer search traffic most often, by far, is YouTube, according to Similarweb.
Not everyone thinks the web is in decline — on the contrary, it is in 'an incredibly expansionary moment', argues Robby Stein of Google. As AI makes it easier to create content, the number of sites is growing: Google's bots report that the web has expanded by 45 per cent in the past two years.
AI search lets people ask questions in new ways — for instance, taking a photo of their bookshelf and asking for recommendations on what to read next — which could increase traffic.
With AI queries, more sites than ever are being 'read', even if not with human eyes. An answer engine may scan hundreds of pages to deliver an answer, drawing on a more diverse range of sources than human readers would.
As for the idea that Google is disseminating less human traffic than before, Mr Stein says the company has not noticed a dramatic decline in the number of outbound clicks, though it declines to make the number public.
There are other reasons besides AI why people may be visiting sites less. Maybe they are scrolling social media. Maybe they are listening to podcasts.
The death of the web has been predicted before — at the hands of social networks, then smartphone apps — and not come to pass.
But AI may pose the biggest threat to it yet. If the web is to continue in something close to its current form, sites will have to find new ways to get paid for content.
'There's no question that people prefer AI search,' says Mr Gross. 'And to make the internet survive, to make democracy survive, to make content creators survive, AI search has to share revenue with creators.'
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
AU Financial Review
2 hours ago
- AU Financial Review
Is AI making you stupid?
As anybody who has ever taken a standardised test will know, racing to answer an expansive essay question in 20 minutes or less takes serious brain power. Having unfettered access to artificial intelligence (AI) would certainly lighten the mental load. But as a recent study by researchers at the Massachusetts Institute of Technology (MIT) suggests, that help may come at a cost. Over the course of a series of essay-writing sessions, students working with as well as without ChatGPT were hooked up to electroencephalograms (EEGs) to measure their brain activity as they toiled. Across the board, the AI users exhibited markedly lower neural activity in parts of the brain associated with creative functions and attention. Students who wrote with the chatbot's help also found it much harder to provide an accurate quote from the paper that they had just produced.
Sydney Morning Herald
a day ago
- Sydney Morning Herald
Let there be light: Lessons from Melbourne's first apartment high-rise
Here, a two-bedroom apartment is about 80 square metres – about 10 square metres larger than the average modern counterpart. Opat's firm, Opat Architects, has been commissioned by the building's body corporate for projects including upgrades to the lobby, rooftop and, most recently, the lifts. The ground floor once housed a cafe, milk bar and hairdresser, which have since been turned into apartments, but the foyer is full of original design features: the curved stone wall, floor and wall tiles, and copper mailboxes are unchanged. Opat turned a pit that residents termed a 'timber garden of horror' into a centrepiece structure with wood panelling to show off the letterboxes and provide some privacy. Upstairs, the rooftop – from which Luna Park can be seen – had been unusable until Opat installed new flooring and built-in furniture that conceal the building's services. Last year, the ageing lifts were cleverly upgraded to both meet modern accessibility standards and reach the rooftop. Russell Jessop bought his two-bedroom apartment about 20 years ago after he was immediately enamoured by the bay views. He has since taken a keen interest in preserving and promoting the tower's history – Jessop is the self-appointed manager of the building's Wikipedia page. He believes today's apartment developers can learn from many of Edgewater's qualities, such as inviting communal areas, access to open space (Edgewater has a park on one side and the beach on the other) and well-sized apartments brimming with sunlight. 'There needs to be enough space. The apartments are not flashy and over the top, but they are more than enough,' he says. Edgewater's appeal means its apartments sell for above the Melbourne median for comparable units – a one-bedroom sells for about $500,000, while a two-bedroom goes for $900,000. From wider corridors to deeper wardrobes, Opat says Edgewater's apartments are generous in both size and sunlight by Victoria's modern standards. 'There's a lot of cleverness required to fit things in now, but here, there's just provision of what you need without squeezing things in.' Land values, of course, differed in the '60s. But Opat believes there are too many restrictions on what sites can be developed with apartments, even with the government's reforms aimed at loosening planning rules around neighbourhood character and delivering 60 activity centres near transport hubs. Loading He argues that five-storey buildings should be allowed in just about every street in inner and middle Melbourne, like many European cities. Currently, he says, only large wealthy developers can afford to buy sites to build apartments, and they then seek to increase heights and shrink dwellings to make a profit. Opat says that if more spaces are allowed to be more densely developed, there's more competition in the market – and with that, the potential for better apartments. 'What I'm imagining is a lot more severe in change,' he says. He opposes rampant single-storey development of greenfield areas that lack basic infrastructure. 'It's not sustainable by any measure.' He questions why Edgewater Towers remains something of an outlier on St Kilda's foreshore and he feels NIMBY-ism has got in the way of high-density development. Jessop says he would support apartment towers being built on neighbouring sites. Opat says: 'This was acceptable for this site in 1961. We're now 2025. What happened? Something stopped.' A state government spokesman says Labor first introduced minimum apartment design standards in 2017 with requirements for access to natural light and outdoor space. An update to the standards will occur by 2026, four years after a parliamentary inquiry into apartment design published its final report. 'Whether it's building more homes close to trains and trams, slashing stamp duty for off the plan apartments and townhouses, and making it easier to build townhouses across the state – our bold planning reforms are helping deliver more homes where people want to live.' Proud of his home, Jessop hopes that visitors during Open House Melbourne enjoy his building's story. 'Some people think the building is quite scary because it's so different to everything else in the street,' he says. 'By having new people come in, they buy into the [building's] story.'
The Age
a day ago
- The Age
Let there be light: Lessons from Melbourne's first apartment high-rise
Here, a two-bedroom apartment is about 80 square metres – about 10 square metres larger than the average modern counterpart. Opat's firm, Opat Architects, has been commissioned by the building's body corporate for projects including upgrades to the lobby, rooftop and, most recently, the lifts. The ground floor once housed a cafe, milk bar and hairdresser, which have since been turned into apartments, but the foyer is full of original design features: the curved stone wall, floor and wall tiles, and copper mailboxes are unchanged. Opat turned a pit that residents termed a 'timber garden of horror' into a centrepiece structure with wood panelling to show off the letterboxes and provide some privacy. Upstairs, the rooftop – from which Luna Park can be seen – had been unusable until Opat installed new flooring and built-in furniture that conceal the building's services. Last year, the ageing lifts were cleverly upgraded to both meet modern accessibility standards and reach the rooftop. Russell Jessop bought his two-bedroom apartment about 20 years ago after he was immediately enamoured by the bay views. He has since taken a keen interest in preserving and promoting the tower's history – Jessop is the self-appointed manager of the building's Wikipedia page. He believes today's apartment developers can learn from many of Edgewater's qualities, such as inviting communal areas, access to open space (Edgewater has a park on one side and the beach on the other) and well-sized apartments brimming with sunlight. 'There needs to be enough space. The apartments are not flashy and over the top, but they are more than enough,' he says. Edgewater's appeal means its apartments sell for above the Melbourne median for comparable units – a one-bedroom sells for about $500,000, while a two-bedroom goes for $900,000. From wider corridors to deeper wardrobes, Opat says Edgewater's apartments are generous in both size and sunlight by Victoria's modern standards. 'There's a lot of cleverness required to fit things in now, but here, there's just provision of what you need without squeezing things in.' Land values, of course, differed in the '60s. But Opat believes there are too many restrictions on what sites can be developed with apartments, even with the government's reforms aimed at loosening planning rules around neighbourhood character and delivering 60 activity centres near transport hubs. Loading He argues that five-storey buildings should be allowed in just about every street in inner and middle Melbourne, like many European cities. Currently, he says, only large wealthy developers can afford to buy sites to build apartments, and they then seek to increase heights and shrink dwellings to make a profit. Opat says that if more spaces are allowed to be more densely developed, there's more competition in the market – and with that, the potential for better apartments. 'What I'm imagining is a lot more severe in change,' he says. He opposes rampant single-storey development of greenfield areas that lack basic infrastructure. 'It's not sustainable by any measure.' He questions why Edgewater Towers remains something of an outlier on St Kilda's foreshore and he feels NIMBY-ism has got in the way of high-density development. Jessop says he would support apartment towers being built on neighbouring sites. Opat says: 'This was acceptable for this site in 1961. We're now 2025. What happened? Something stopped.' A state government spokesman says Labor first introduced minimum apartment design standards in 2017 with requirements for access to natural light and outdoor space. An update to the standards will occur by 2026, four years after a parliamentary inquiry into apartment design published its final report. 'Whether it's building more homes close to trains and trams, slashing stamp duty for off the plan apartments and townhouses, and making it easier to build townhouses across the state – our bold planning reforms are helping deliver more homes where people want to live.' Proud of his home, Jessop hopes that visitors during Open House Melbourne enjoy his building's story. 'Some people think the building is quite scary because it's so different to everything else in the street,' he says. 'By having new people come in, they buy into the [building's] story.'
