Moore’s Law, an observation, really, was formulated in 1965 by Intel co-founder Gordon Moore, holding that the number of transistors on a chip doubles roughly every year. In 1975, he adjusted the formula to every two years. It held true for decades and helps explain a lot about today’s world, including why the price-performance of smartphones and other digital devices improved spectacularly, especially compared with many other kinds of goods. Transistors once sold for as much as $150 each. In 2019, a typical Intel microprocessor contained five billion of them—each costing $0.0000001.
But with transistor size now so small, the steady predictable progress that Moore’s Law described has come to an end, some say, as certain efforts approach the limits of economics and physics. Other kinds of semiconductor innovation, closely tied to advances in deep learning and other forms of AI, are under way. But, it’s no longer possible to describe the next stages of chip innovation that predict the future in such a clear way, readily grasped by the general public.
“Those days are gone, right?” said Bill Dally, chief scientist at semiconductor powerhouse Nvidia, which designs the chips most in demand for artificial intelligence. Nvidia is known, among other things, for graphics processing units, a kind of chip that employs many cores that make simultaneous calculations. GPUs are greatly in demand for use with AI, especially neural networks that have many dense layers.
While the advances described by Moore’s Law aren’t racing ahead as they once did, researchers at Nvidia, Google DeepMind and elsewhere are pushing ahead on a range of fronts.
“Deep learning was powered by hardware. We feel great pressure to continue,” said Dally, a former chairman of the computer science department at Stanford University, where he remains an adjunct professor.
“It’s getting harder, but we still have good ideas…out to like four years out, we’ve got a pretty good idea of where the performance is coming from. And we have a lot of exploratory efforts to answer the question about where the performance is going to come from after that,” said Dally, who leads a team of 300 researchers with Ph.D.s.
Nvidia’s advanced Hopper architecture for data center chips was launched in 2022 and managed to pack more transistors onto the chip, but the chip cost more than prior versions and the individual transistors cost more as well, according to Dally. “The smaller transistors aren’t really helping us very much,” said Dally, adding, “we’re no longer getting a reduction in the cost of transistors as we move from one generation to the next.”
Nonetheless, the price performance of Hopper advanced, given other kinds of innovations that exist outside of Moore’s Law. Dally said he believes that the path toward more innovation is clear for the next four years at least, which means that the hardware improvements closely associated with the growing power of AI should continue as well, with implications for a huge range of sectors, from medical science and healthcare to finance and e-commerce.
This work draws on concepts such as sparsity, or rounding “little” numbers down to zero, thereby reducing the amount of multiplication that must be performed by a chip.
Eight years ago, Dally and Song Han, now at MIT, and others published a paper at the NeurIPS conference that showed for a lot of neural networks, they could make 90% of the numbers zero and it didn’t change the accuracy of the outcome.
“We’re still catching up with how to exploit that with better hardware,” Dally said.
Another area of innovation involves number representation, in which a number is represented in various ways, such as a string of ones and zeros. If this can be done with shorter strings—say eight instead of 16 or 32 units or bits—the amount of work demanded of the chip during processes such as multiplication is reduced accordingly, according to Dally. That boosts the overall power and efficiency of the chip.
Nvidia and Google have been working on customizing chips for AI. That has allowed them to scale AI chip performance faster than Moore’s Law, even though it’s harder and harder to put more transistors on a chip. In 2020, Wall Street Journal columnist Christopher Mims coined the term Huang’s Law after Nvidia co-founder and Chief Executive Jensen Huang, which stated that the silicon chips that power AI more than double in performance every two years.
There is, however, a fundamental discrepancy in that AI algorithms evolve every few months, while it takes two-to-three years to design a chip.
By employing advanced AI, it may be possible to significantly reduce the time and cost that go into chip design. Some researchers think AI could reduce the design process from a matter of years to a matter of days. On the software side, better scheduling of resources and more optimal code could lead to improved performance on existing chip designs.
There is also a more subtle consequence of slow and costly chip design: it requires AI researchers to create algorithms for existing chips, which in turn may limit the span of AI algorithms they envision, according to Olivier Temam, director, data center and chip research, Google DeepMind. If researchers can more easily and cheaply create chips adapted to novel AI algorithms, it might unleash more creativity in that field.
Says Temam: “The latest advances in AI-powered chip design combined with ongoing explorations, suggest we may be able to automate chip design one day, eventually leading to massive productivity gains, which could transform the industry.”
Write to Steven Rosenbush at [email protected]
Milestone Alert!Livemint tops charts as the fastest growing news website in the world 🌏 Click here to know more.