How hidden humans help 'self-driving' cars navigate city streets

Las Vegas: In places like San Francisco, Phoenix and Las Vegas, robot taxis are navigating city streets without a driver behind the steering wheel. Some don't even have steering wheels.

But some of the cars are sometimes guided by someone sitting at a command center in Foster City, California, operated by Zoox, a self-driving car company owned by Amazon. Like other robot taxis, the company's self-driving cars sometimes struggle to drive themselves, so they get help from human technicians sitting in a room about 500 miles away.

Inside companies like Zoox, this kind of human assistance is taken for granted. Outside these companies, few realize that autonomous vehicles are not completely autonomous.

For years, companies avoided mentioning the remote assistance provided to their self-driving cars. The illusion of complete autonomy helped draw attention to their technology and encourage venture capitalists to invest the billions of dollars needed to build increasingly effective autonomous vehicles.

"There is a Wizard of Oz flavor to this," said Gary Marcus, an entrepreneur and an emeritus professor of psychology and neural science at New York University who specializes in artificial intelligence and autonomous machines.

If a Zoox robot taxi encounters a construction zone it has not seen before, for instance, a technician in the command center will receive an alert -- a short message in a small colored window on the side of the technician's computer screen. Then, using the computer mouse to draw a line across their computer screen, the technician can send the car a new route to follow around the construction zone.

"We are not in full control of the vehicle," said Marc Jennings, 35, a Zoox remote technician. "We are providing guidance."

As a Zoox car approached an emergency scene one recent evening in San Francisco, it was unable to navigate around a fire truck in front of it.

The car alerted a remote technician 35 miles away in Foster City. The technician could see video feeds from multiple cameras installed on the car.

From a dashboard on a computer screen, the technician also saw a graphical bird's-eye view of what the car was encountering on the road. In the corner of the dashboard, text alerted the technician that the car was stalled and in need of human intervention: "Hero is not making progress 6:22:07 PM."

To route the car around the fire truck, the technician used a computer mouse to set a new path for the car -- a line of so-called waypoint dots. As the vehicle began to move along its new path, the technician continued to lay breadcrumbs it should follow.

But soon the technician realized there was an ambulance behind the fire truck, something that was obscured from the car's cameras until it had moved far enough to the left. As the technician adjusted the path on the fly, the car navigated the new course -- eventually returning to its lane and to autonomous driving.

As companies like Waymo, owned by Google's parent company, Alphabet, and Cruise, owned by General Motors, have begun to remove drivers from their cars, scrutiny of their operations has increased. After a series of high-profile incidents, they have started to acknowledge that the cars required human assistance.

While Zoox and other companies have started to reveal how humans intervene to help the driverless cars, none of the companies have revealed how many remote assistance technicians they employ or how much it all costs. Zoox's command center holds about three dozen people who oversee what appears to be a small number of driverless cars -- two in Foster City and several more in Las Vegas -- as well as a fleet of about 200 test cars that still have a driver behind a steering wheel.

When regulators ordered Cruise to shut down its fleet of 400 robot taxis in San Francisco after a woman was dragged under one of its driverless vehicles, the cars were supported by about 1.5 workers per vehicle, including remote assistance staff. Those workers intervened to assist the company's vehicles every 2.5 to 5 miles, according to two people familiar with its operations.

The expenses associated with remote assistance are one reason robot taxis will struggle to replace traditional ride-hailing fleets operated by Uber and Lyft. Though companies like Zoox are beginning to replace drivers, they still pay people to work behind the scenes.

"It may be cheaper just to pay a driver to sit in the car and drive it," said Thomas W. Malone, a professor at the Massachusetts Institute of Technology's Center for Collective Intelligence.

Waymo and Cruise declined to comment for this story.

Whereas Waymo and Cruise use traditional cars retrofitted for self-driving, Zoox is testing a new kind of vehicle in Foster City, just south of San Francisco, and in Las Vegas, not far from the Strip.

These robot taxis are not yet available to the public. But Zoox recently allowed New York Times reporters to take a test ride.

Inside, there is no place for a driver. There is no driver's seat, no steering wheel, no dashboard and no gas pedal. The car seats four people. And all four are passengers. They sit on two bench seats that face each other, like two couches in a living room.

When you take a ride, your only interaction with the car is through a touch screen that opens and closes the door. If you like, you can adjust the air conditioning or play some music. But that's about it.

Unlike passengers in other self-driving cars, you do not see pedals moving up and down or a steering wheel turning on its own. In theory, this is a more efficient way of building a self-driving car. To the rider, it seems as if the car is handling everything on its own.

After testing the vehicles with Zoox employees, their family members and friends, the company plans to unveil a public service this year. But this robot taxi service, like all others, will lean on human assistance.

In Foster City, the company operates what it calls a "Fusion Center," where employees monitor robot taxis operating both locally and in Las Vegas, several hundred miles away. From their computer screens, these workers can track live feeds of the road from cameras installed on the cars as well as a detailed overhead view of the car and its surroundings, which is stitched together using data streaming from array of sensors on the vehicle.

These workers can provide verbal assistance to riders via speakers and microphones inside the cars. They can also assist the cars if they encounter a scenario they cannot handle on their own.

"These are situations that don't necessarily fit the mold," said Jayne Aclan, who oversees a team of Zoox technicians that provide cars with remote assistance.

Self-driving cars can reliably handle familiar situations, like an ordinary right turn or lane change. They are designed to brake on their own when a pedestrian runs in front of them. But they are less adept in unusual or unexpected situations. That's why they still need the humans in the control room.

But even though self-driving cars can lean on this remote human assistance, they still make mistakes on the road.

On a recent afternoon in Las Vegas, a Zoox robot taxi rolled down a multilane road a few miles away from the Strip. On the other side of the street, a yellow Clark County fire vehicle approached with its sirens on.

As other cars on the road stopped for the emergency vehicles, the Zoox car continued driving. That prevented the fire truck from turning left until after the Zoox car passed. It was a reminder that driverless vehicles sometimes struggle to respond to so-called edge cases, like passing emergency vehicles.

After reviewing the incident, Zoox indicated that its cars struggled to recognize the fire trucks because they were yellow, not red. "We continue to test and refine our driving software," the company spokesperson Whitney Jenks said.

It will also continue to lean on human assistance.

"We think that computers should be able to replicate humans and replace humans in all ways," Malone said. "It is possible that might happen. But it hasn't yet."