a day ago
Caller ID destroying our understanding of the jobs market
Getting households to pick up the phone is just one of many obstacles facing governments who have to collect data on everything. PHOTO: UNSPLASH
LONDON - The UK's Office for National Statistics (ONS) used to knock on doors and ask people if they were employed, as a traditional way of compiling the country's labour market data.
But when Covid-19 forced social distancing in 2020, door-knocking was out. At one point during the pandemic, the survey was reliant on how many phone numbers it could find for a representative sample – and also on people actually answering their mobile phones when an unknown number called. Only a quarter of Britons say they answer such calls, with most wary of scams. Fewer that half of UK households have a landline, and those numbers can be difficult to find.
Response rates plunged. By 2023, the data had become so patchy the ONS was forced to suspend its unemployment reading as it desperately sought to repair a key release that helps inform government policy, influences interest rates and drives billions of dollars of investment decisions. An ensuing crisis of confidence in Britain's data ended with the resignation of national statistician Ian Diamond in May, following fierce criticism from politicians and senior central bankers.
Getting households to pick up the phone is just one of many obstacles facing government agencies tasked with collecting numbers on everything from joblessness to agricultural output. Just as cellphone users swipe away unwanted callers, people screen visitors with video doorbells and an increasing number live in apartment blocks impenetrable to data collectors, compounding a collapse in survey responses. The pandemic exacerbated most of these trends.
That's left statisticians from Australia to Britain and the US scrambling to update collection methods stuck in the 20th century without introducing new 21st century biases. The solution that agencies and pollsters, from the ONS to the Bureau of Labor Statistics in the US, have zeroed in on: online surveys.
These polls are less costly than telephone or face-to-face surveys, more convenient for respondents, and statistics agencies are using AI to target non-responders. Yet as agencies are finding, the big switch in economic data-collecting is throwing up new problems for the indicators global investors and policymakers rely on.
'There's no magic mode,' said Steve MacFeely, chief statistician at the Organisation for Economic Cooperation and Development. 'They're well aware of the risk and are doing it because they have no other choice. It's the grim reality of official statistics today.'
Barrage of spam
Problems with the new wave of online surveys include fresh biases in the data, confusion over questions that a computer cannot address, gaining trust amid a barrage of spam and fraud in people's inboxes, and the fact that households may be just as likely to ignore an email with an online survey as a phone call.
'What you gain in coverage/response, you possibly lose in measurement quality or content features,' said Eric Harrison, deputy director of the European Social Survey.
Problems with the UK's labour market figures serve as a cautionary tale.
Early results of the UK's online questionnaire showed the new online survey was producing significantly different readings from the existing survey that relies on telephone calls and knocking on doors, people familiar with the situation said.
One potential issue is that respondents aren't able to ask an interviewer to clarify a question they don't understand. The ONS is looking at an AI chatbot that could help.
'Moving the survey online has been challenging due to the survey length and complexity meaning many households didn't finish it,' an ONS spokesperson said. 'We've tested a shorter version that's giving improved and more complete responses and this new version will go live in the field next month.'
Another key risk from online surveys is that it creates new unknown biases that statisticians have to counteract. In the UK, the existing labour force survey struggled to get enough young workers to answer. For an online questionnaire, it could be the opposite problem.
'If you were to do a face-to-face survey, and you were to do a telephone or an online labor force survey, you'd get different results from all three,' said Mr MacFeely from the OECD.
In the US, Joanne Hsu, director of consumer surveys at the University of Michigan, believes that being behind the computer screen may also skew how people respond to economic surveys.
The Michigan survey's closely watched figures on economic sentiment and inflation expectations in the US fully transitioned to a web-only mode in 2024 after seven years of testing the new responses alongside its existing phone-based method. While it helped to double the survey's reach to nearly 1,500 responses, Ms Hsu noticed that respondents are more likely to give extreme answers on the web – especially for inflation expectations, which recently surged to the highest in three decades.
'People might think twice about giving any sort of extreme value to an interviewer,' she said.
Former US Bureau of Labor Statistics commissioner William Beach said that US data could soon be in the same state as Britain's moribund labour statistics without action to modernise its surveys.
Labor force surveys 'are dying, they're decaying, they're in very serious trouble,' Mr Beach recently said on Bloomberg's Odd Lots podcast. 'Unless we modernize that survey, we will see a time when we will be like the British, unable to publish portions of it that just don't have sufficient sample for statistical release.' BLOOMBERG
Join ST's Telegram channel and get the latest breaking news delivered to you.
