Toyota spends $10 billion a year on research, more than any other automaker except Volkswagen. That pays for endless incremental improvements in everything from lithium batteries to seatbelt design, but such tweaks may not be enough anymore if Toyota is to remain the world’s top seller of cars.

The development of autonomous vehicles now threatens to change the very essence of driving. In the race to develop that technology, Toyota lags behind several of its fellow carmakers and Silicon Valley upstarts such as Google and Tesla Motors. It’s possible that a generation from now everything from roadway design to driver certification will be radically reshaped by the ubiquity of semi- or fully autonomous vehicles, and carmakers without the requisite technology will be as imperiled as the sellers of silver-halide film in the age of the digital selfie.

Determined to make up for lost time, Toyota’s 60-year-old CEO, Akio Toyoda, is spending $1 billion for a new Toyota Research Institute with offices in Michigan, Silicon Valley, and Cambridge, Massachusetts, that will focus on autonomous cars and robotics. He has recruited Gill Pratt, a top robotics researcher, to run the institute, giving him authority to hire hundreds of engineers and scientists. At the same time, Toyota is striking up partnerships with Stanford, the University of Michigan, and MIT to rethink cars’ capabilities, even if provocative new approaches might take a decade or longer to show up in dealer showrooms.

It’s clear that Toyota, like most established carmakers, isn’t making an all-out bid to match Google’s efforts to build fully autonomous vehicles. Instead, Toyota envisions drivers and software sharing control for years to come. Pratt is championing “guardian angel” technology that could find the best evasive strategies in an instant if trouble looms.

Similarly, artificial-intelligence researchers at Stanford who are working with Toyota are testing out inward-looking cameras that could help assess drivers’ alertness. If drivers get drowsy or stop paying attention to the road, then automated safety systems could help keep the car safe while nudging the driver to get back on task.

As cars take on more and more tasks, just how smart can they get? Fei-Fei Li, a Stanford computer science professor who is heading her department’s $25 million alliance with Toyota, says her team is applying a wide range of AI techniques to driving-related challenges. As she cheerfully acknowledges, “Our work might be relevant to the cars of 2018, or 2028, or anywhere in between.”

One area of interest: defensive driving. Is it possible to teach a car’s software to anticipate trouble that could emerge from an obscured side street, a wobbling bicycle, or an angry motorist who is switching lanes in a dangerous way? Teams of Stanford researchers are testing out approaches. One initiative, led by John Duchi, an assistant professor of statistics and electrical engineering, is starting with known hazards, such as erratic bicyclists, and then trying to build prediction software that could make smart decisions in similar situations. Another team, led by Li, is relying on 3-D vision and pattern recognition to identify high-risk groupings. These include pedestrians staring at smartphone screens, or children playing catch near a roadside. Put a human driver behind the wheel, and it’s easy to distinguish an alert pedestrian from a badly distracted one. Ask software and sensors to be equally discerning, and “it’s a very difficult research problem,” Li says.

A few years ago, Li developed software that could almost unfailingly identify objects in photos, even if they were in odd poses or in front of confusing backgrounds. Now she is building on those techniques to analyze roadway photos. Her goal: to ensure that a car’s software can detect the difference between a pedestrian at the curb making eye contact with drivers and a pedestrian listening to music through earbuds and gazing downward at a smartphone.

Li’s group is also creating ways that self-driving cars can share information as instantaneously as possible. On highways, for example, it should be possible to safely compress the spacing between cars, helping traffic flow more smoothly, as long as vehicles at the back of a convoy can be apprised of any surprises that the front car has already identified. Even in city traffic, crashes could be avoided if cars could instantly communicate with one another about hazards that might be invisible to one vehicle but easily recognizable from a different perspective.

Getting data

In 2012, when Google was testing Toyotas that it modified to drive themselves, the automaker turned down an offer from Google to coöperate on the technology because it was reluctant to share manufacturing know-how. Google has instead taken steps toward an alliance with Ford. Even as recently as 2014, CEO Toyoda said he wasn’t inclined to take autonomous technology seriously until a self-driving car could beat the best humans in a 24-hour test on a top German racetrack.

By now, Google and several automakers have built up large stockpiles of video and sensor data from years of testing autonomous cars and selling models with some autonomous features, such as lane-departure warnings and blind-spot detection. Having less information to feed into their machine--learning systems could put Toyota’s software researchers at a disadvantage.

Both Pratt and Li have been calling for car companies to share data from autonomous vehicles in the belief that pooled knowledge will help all competitors make faster progress and gain public trust. After all, self-driving cars could be far safer. More than six million motor-vehicle crashes take place each year in the United States, killing about 33,000 people. As much as 90 percent of those accidents can be traced to human error. Still, it’s unlikely that data-rich companies will want to relinquish control of their own hard-won knowledge that—for now, at least—provides them with a competitive edge.

Raj Rajkumar, co-director of the General Motors/Carnegie Mellon autonomous-driving lab, puts GM, Nissan, and the German Big Three (Daimler, VW/Audi, and BMW) in the upper echelon of global car companies moving toward some degree of autonomous driving. “Toyota seems to be lagging behind,” he says. “But with the creation of its research institute, it might catch up quickly.” One of Pratt’s first hires at the institute was a former Google robotics director, James Kuffner, as chief technology officer.

In the university partnerships, Toyota isn’t the only car company that could benefit; other manufacturers will eventually be able to consult published findings, Li says. Still, while the pieces come together, Toyota will enjoy special access and collaboration. And car technology’s long adoption curve may give Toyota the time it needs. Lots of battles still lie ahead in winning regulatory approval and customer loyalty, regardless of whose early technology is most promising. History shows that breakthrough technologies such as airbags and advanced transmissions can take 20 years to gain mass acceptance after their marketplace debut.

Toyota also figures to move faster now that its boss has repudiated his earlier doubts. In May the CEO urged all his employees to embrace the “momentous change” associated with automated driving and robotics. Such new technologies will be as transformative to the company, Toyoda vowed, as was his own grandfather’s decision in 1930 to create a motor-car division within what was then a small loom-making
company.