The next AI explosion will be defined by the chips we build for it
Hardware design, rather than algorithms, will help us achieve the next big breakthrough in AI. That’s according to Bill Dally, Nvidia’s chief scientist, who took the stage Tuesday at EmTech Digital, MIT Technology Review’s AI conference. “Our current revolution in deep learning has been enabled by hardware,” he said.
As evidence, he pointed to the history of the field: many of the algorithms we use today have been around since the 1980s, and the breakthrough of using large quantities of labeled data to train neural networks came during the early 2000s. But it wasn’t until the early 2010s—when graphics processing units, or GPUs, entered the picture—that the deep-learning revolution truly took off.
“We have to continue to provide more capable hardware, or progress in AI will really slow down,” Dally said.
Nvidia is now exploring three main paths forward: developing more specialized chips; reducing the computation required during deep learning; and experimenting with analog rather than digital chip architectures.
Nvidia has found that highly specialized chips designed for a specific computational task can outperform GPU chips that are good at handling many different kinds of computation. The difference, Dally said, could be as much as a 20% increase in efficiency for the same level of performance.
Dally also referenced a study that Nvidia did to test the potential of “pruning”—the idea that you can reduce the number of calculations that must be performed during training, without sacrificing a deep-learning model’s accuracy. Researchers at the company found they were able to skip around 90% of those calculations while retaining the same learning accuracy. This means the same learning tasks can take place using much smaller chip architectures.
Finally, Dally mentioned that Nvidia is now experimenting with analog computation. Computers store almost all information, including numbers, as a series of 0s or 1s. But analog computation would allow all sorts of values—such as 0.3 or 0.7—to be encoded directly. That should unlock much more efficient computation, because numbers can be represented more succinctly, though Dally said his team currently isn’t sure how analog will fit into the future of chip design.
Naveen Rao, the corporate vice president and general manager of the AI Products Group at Intel, also took the stage and likened the importance of the AI hardware evolution to the role that evolution played in biology. Rats and humans, he said, are divergent in evolution by a time scale of a few hundred million years. Despite vastly improved capabilities, however, humans have the same fundamental computing units as their rodent counterparts.
The same principle holds true when it comes to chip designs, Rao said. Any chip—whether specialized or flexible, digital or analog, optical or otherwise—is simply a substrate for encoding and manipulating information. But depending on how that substrate is designed, it could be the difference between the capabilities of a rat and a human.
Insects, like rats, he said, are also built with the same fundamental units as humans. But insects have fixed architectures whereas humans have more flexible ones. Neither one, he argued, is superior to the other, but they clearly evolved to suit different purposes. Insects can likely survive a nuclear war, while humans have much more sophisticated capabilities.
Again, those principles can be applied to chip design. As we bring more smart devices online, it won’t always make sense to send their data to the cloud in order to be processed through a deep-learning model. Instead, it may make sense to run a small, efficient deep-learning model on the device itself. This idea, known as “AI on the edge,” could benefit from specialized, fixed chip architectures that are more efficient. Data centers that power “AI on the cloud,” on the other hand, would run on fully flexible and programmable chip architectures, to handle a much broader spectrum of learning tasks.
Rao noted that whatever chip designs Intel and Nvidia decide to pursue, the effect on the evolution of AI will be significant. Throughout history, individual civilizations evolved in very different ways because of the unique materials at their disposal. Likewise, the operations that Intel and Nvidia make easier through different chip designs will heavily influence the kinds of learning tasks the AI community will pursue.
“We’re in this rapid Precambrian explosion [for chip architectures] right now,” Rao said, “and not every solution is going to win.”
The inside story of how ChatGPT was built from the people who made it
Exclusive conversations that take us behind the scenes of a cultural phenomenon.
AI is dreaming up drugs that no one has ever seen. Now we’ve got to see if they work.
AI automation throughout the drug development pipeline is opening up the possibility of faster, cheaper pharmaceuticals.
GPT-4 is bigger and better than ChatGPT—but OpenAI won’t say why
We got a first look at the much-anticipated big new language model from OpenAI. But this time how it works is even more deeply under wraps.
The original startup behind Stable Diffusion has launched a generative AI for video
Runway’s new model, called Gen-1, can change the visual style of existing videos and movies.
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.