As Robotaxi Rides Begin, We Still Don't Know the Mystery of Tesla's Human Helpers

WIRED

a day ago

Neither the US federal government nor the City of Austin will say how teleoperations, self-driving's critical safety feature, will be used in the service launching in Austin in just a matter of days. Photograph: Sjoerd vanSelf-driving vehicle developers don't usually love talking about 'teleoperation'—when a human guides or drives robot cars remotely. It can feel like a dirty secret. Shouldn't an autonomous vehicle operate, well, autonomously ?
But experts say teleoperations are, at least right now, a critical part of any robot taxi service, including Tesla's Robotaxi. The tech, though impressive, is still in development, and the autonomous systems still need humans to guide them through less-common and especially sticky road situations. Plus, a bedrock principle of safety engineering is that every system needs a backup—doubly so for new robotic ones that involve two-ton EVs driving themselves on public roads.
And yet, just days out from Tesla's launch of its long-awaited (and much delayed) Robotaxi service in Austin, Texas, the public still doesn't know much at all about its teleoperations systems. Tesla has posted a job related to teleoperations where it states the role will be responsible for developing the application "that our Remote Operators use to interface with our cars and robots', an application where these operators will be 'transported into the device's world using a state-of-the-art VR rig that allows them to remotely perform complex and intricate tasks.'
Alarmingly, several government spokespeople—representing the city of Austin, the state of Texas, and the US' top road safety regulator—didn't respond to questions about Tesla's teleoperations. Indeed, Austin and the Texas Department of Transportation referred all our questions about Tesla technology to the company itself. Tesla, which disbanded its public relations team in 2020, didn't respond to WIRED's questions.
Last month, the National Highway Traffic Safety Administration, the country's road safety watchdog, wrote a letter to Tesla posing questions about, among other things, how or if Tesla planned to use teleoperations. How will its human staff be expected to monitor, supervise, or even intervene when its systems are on the road? The government asked the company to respond by June 19, which will be after the service supposedly launches on June 12, according to reporting from Bloomberg earlier this month. NHTSA repeatedly would not respond to WIRED's inquiries into what it knows about Tesla's teleoperations.
The Los Angeles Times reported that humans used teleoperations to manipulate the robot Optimus during a 'Cybercab' debut event in Los Angeles, and when Optimus showed off its new hands a month later, catching a tennis ball in mid-air, an engineer for the company acknowledged that humans similarly used teleoperations. The company also has a permit to test autonomous vehicles in California with a driver behind the wheel. The state has much stricter rules than Texas, and requires some kind of 'communication link' between testing vehicles and remote operators, so it's likely the company has some kind of system.
While not shedding any light on exactly how Tesla's teleoperations will work in the city, Austin Transportation and Public Works spokesperson Cristal Corrales wrote in an email: 'The City works with AV [autonomous vehicle] companies before and during deployment to obtain training for first responders, establish expectations for ongoing communication and share information about infrastructure and events.' Texas Department of Transportation spokesperson Laura Butterbrodt said in an emailed statement: 'Texas law allows for AV testing and operations on Texas roadways as long as they meet the same safety and insurance requirements as every other vehicle on the road.'
Bedeviling this Robotaxi mystery is the fact that the autonomous vehicle industry hasn't coalesced around a definition for 'teleoperations.' So as Tesla watchers await the Austin service, it's worth understanding a bit more about these 'teleoperations,' and how they work.
The interior of Tesla's autonomous Cybercab taxi, showing the complete lack of physical controls—no steering wheel or pedals. Photograph:A Little Help From My Friends
It's worth defining some terms. What self-driving car developers usually call 'remote operations' refer to a few different sorts of human jobs. There are, first, the operators who deal with other humans. These are people trained to interact with autonomous taxi riders when they have questions or need assistance in an emergency. Alphabet's Waymo, the undisputed leader in self-driving, has a big 'Support' button on its in-car passenger screens, which can connect riders with these folks. These operators can also be people trained to interact with law enforcement or emergency responders when they need assistance.
Then there are operators who deal with the autonomous system. Some of these people may work in 'remote assistance.' More confusion: Self-driving car developers give these tasks different names and titles. Amazon subsidiary Zoox uses 'teleguidance'; the self-driving truck developers at Aurora like 'teleassistance'; Tesla appears to stick to 'teleoperation'; Waymo calls these workers 'fleet response agents.' Whatever they're called, these humans are meant to guide the autonomous system when it needs help. A car might alert people, for example, when it has encountered a road block, like construction equipment; the remote assistant might suggest a lane change, or a turn, or even a quick dip onto a road's shoulder to get around it.
These human assistants can also sometimes reroute an autonomous vehicle if its planned itinerary no longer makes sense—maybe a road is unexpectedly closed off for a street fair. They can also sometimes help the car identify objects that have, for whatever reason, confused its sensors: a plastic bag flapping in the wind, or a traffic light (red or green?).
Remote assistance should be a part of every safe self-driving vehicle program, says Philip Koopman, a professor at Carnegie Mellon University who studies autonomous vehicle safety. 'The technology is not there for them to be able to handle everything, and that's OK,' he says. Having humans operate in the background of autonomous systems, then, isn't 'cheating' at self-driving. It's understanding the limits of today's technology—and what it takes to run a profitable business based on self-driving cars.
Still, some remote assistance programs are safer than others, Koopman says. Some of that comes down to 'triggers,' or how the system knows it needs help, and turns to its human overlords. Is it the human assistant's job to notice that the vehicle is stuck—or about to smash into something? Or is the onus on the vehicle to ask for aid? The safer bet, he says, is to train the technology to know when it needs intervention, rather than relying on the vigilance of the human auxiliaries.
Asking for help can be very hard for people—but it might be harder for robots. That's because autonomous systems must automatically provide the humans with everything they need to know to assist: what happened; where the vehicle is in space; and if there's anything around it. Only then can the humans guide the robots.
A Tesla Cybercab prototype at a Tesla store in San Jose, California, in November 2024. Photograph: DavidHumans in the Loop
If that sounds tough, even tougher—and more controversial—is 'remote driving.' This is what most people probably think of when they hear 'teleoperations': Someone far away from the supposed self-driving car, behind their own steering wheel or joystick, piloting it like a long-distance RC car.
Remote driving has even more technical challenges. Ben Shukman should know—as an engineer at a startup called Phantom Auto that focused on remote driving, he believes he was the first to do it on public roads. The first issue is connectivity. 'Your ability to drive a car without being in the car is only as stable as the internet connection that connects you to it,' he says.
But anyone who has called a friend on a long drive knows that networks drop in and out as you move in space. There are technical ways to knit together networks, but those aren't foolproof. This leads to big issues with latency. So imagine the worst case scenario: A robot car needs help navigating around an accident on a highway, a remote driver gets it moving, and then … the connection dies.
Another challenge in remote driving: Helping drivers understand the experience of driving without actually being in the car. It's hard to understand how quickly the momentum is shifting, or how hard you're braking if you're not inside the car. Shukman says it's possible to build a user interface that gives remote drivers a sense of what it's like on the road, but this takes thought and time.
For these reasons, Shukman says, remote driving is less than safe in environments where vehicles are moving quickly in unpredictable environments, even above a handful of miles per hour. Today, the technique is mostly used in public to get delivery robots out of jams. Those move so slowly that a few milliseconds of dropped connection likely won't spell disaster. The startup where he worked, Phantom Auto, eventually pivoted to operating forklifts remotely in warehouse settings. (It shut down last year, but its founders are sticking with its thesis: Their new startup keeps humans 'in the loop' by building a platform that allows people to easily intervene to help in AI search.) The Tesla Question
If Tesla does make good on its promise to start its Robotaxi service in Austin this month, how will it handle teleoperations? A Morgan Stanley research note from its head of global autos and shared mobility research Adam Jonas claims the service will be heavily teleoperated, though Tesla has not confirmed any of this.
The job posting for software engineers working in teleoperation on 'Optimus & Robotaxi' explicitly says the company's remote operators will be 'transported into the device's world using a state-of-the-art VR rig' that will supposedly let them remotely execute intricate tasks requiring some form of human involvement. This sounds more like remote driving— direct intervention with the driving task—than remote assistance.
If Tesla's remote operators are meant to closely monitor its Robotaxis' systems, 'I think it's going to be very difficult to expand beyond a few vehicles and a small area,' says Koopman, the professor. Or at least, to do it safely.
In fact, that's what CEO Elon Musk has said—that Tesla's Robotaxi launch will start with just 10 to 20 cars, and expand from there. Maybe there will be some robots. But the better questions are what the humans supporting them will be doing—and whether they'll be doing it safely.