Compute Power Challenge: The Race To Accelerate Data Movement

Forbes

2 days ago

By Dr. Steven Woo, fellow and distinguished inventor at Rambus.
Exciting developments in AI inference—like chain-of-thought prompting, which breaks down large, complex questions and prompts into smaller steps mimicking human reasoning—are opening the gates to the highest quality answers and transparency in the logical inference process of AI.
However, this leap in computing puts even more pressure on already-strained computing resources. New AI infrastructure packs more onto each chip, increasing complexity and power. Recent research from Polytechnique Insights estimates that by 2027, AI servers could consume 85 to 134 terawatt-hours of electricity every year.
The challenges of faster and 'smarter' AI are not new, but the level of computing demand has forced a rethinking of chip and server architecture.
Data Movement: A Critical Limiter
Take, for example, OpenAI's comments on the compute horsepower required for its high-quality image generation tools "melting" graphics processing units. The complexity and speed at which AI images were created led to forced temporary rate limits and longer processing times, a prime example of the relentless demand for more compute power in AI systems.
As the demand for faster computation grows, the fast movement of data back and forth between compute engines and memory creates a litany of problems in terms of cooling, power delivery and reliability.
Faster processing demands faster data delivery, or the benefits of better computation are unrealized. The industry is faced with reworking the infrastructure: moving data closer to the memory, creating shortcuts to reduce the distance data travels and preserving energy and power.
A Turning Point For GPU Architecture
At the chip level, the industry is investigating stacking.
The stacked configuration allows more bits and larger capacity in the same footprint, reducing the distance and, in turn, reducing data movement power. As the chips are stacked, wires are added to create the connections between the stacked die and the processor. This creates a much more efficient flow of data.
However, the process of introducing more wires creates a new set of power and thermal management issues as the system does more work in a smaller volume.
Raising The Voltage To Save The Cabling
Another growing pain point for the industry is the amount of electrical current coursing through the cables to power AI systems. Data centers have traditionally used 12-volt power distribution to supply power to servers, and as power consumption dramatically rose, so too did the amount of current.
Copper cables are the industry standard; these wires have a small amount of resistance. When current passes through them, they heat up, and some of the power is lost in 12-volt systems. The required current has grown so high that the heat that would be generated can potentially melt some existing cables.
To reduce the current, industry leaders are switching power distribution infrastructure to 48 volts. On a static basis, this cuts current levels by a factor of four for the same power. But given the projected growth in AI, power demands will continue their heady rise, and the industry is beginning to discuss infrastructure at much higher voltages, including 400 volts and higher.
One challenge is that the GPUs and other chips for AI use very low voltages, on the order of 1 volt or less. A conversion is required to step down the power from 12, 48 or higher voltage needed by chips. Each voltage conversion comes with some energy loss, so the goal is to accomplish this step-down in as few conversions as possible. Further, the conversion should be done as close as possible to the chips to minimize the distance, which in turn reduces power losses and heat generated when moving the stepped-down power to the components that need it.
Power management integrated circuits (PMICs) will be an increasingly critical part of the compute power distribution infrastructure in the future. These chips convert from the higher voltages used to move power within a data center to the low voltages required by the GPUs and other computing chips. PMICs can be placed near the chips they serve, keeping the distance the current travels to a minimum.
The Heat Is On: Liquid Cooling To The Rescue
Heat is an inevitable byproduct when chips are at work. As chips heat up, expansion occurs, leading to mechanical challenges, including cracking and stress-induced failures from thermal cycling. And perhaps most importantly, the risk of data loss increases as chips become overheated. Temperature sensors can mitigate these issues by alerting host processors when heat flares. Even so, sometimes these systems are not enough.
Traditional cooling systems, which use forced air blowing over heatsinks, are not designed to handle the power densities of AI servers. And with cooling and ventilation systems accounting for 30% to 55% of data center power consumption, efficiency is becoming more critical.
In the past, systems used chilled air to help with thermal regulation, but with rising power consumption and increased compute density, this solution is no longer sufficient. More recently, the industry has turned to liquid cooling. Compared to previous solutions, liquid cooling offers a much higher capacity to transfer heat compared to air, making it a good solution for current and future AI systems.
But there are tradeoffs: Modern liquid cooling systems can cost between $1,000 to $2,000 per kW cooled. Liquid cooling machinery is also much heavier than traditional systems, requiring data centers to install infrastructure, including pumps and, in some cases, stronger flooring to support the weight of the machinery and liquid.
As more data centers for AI servers are built, we'll likely see a move to this form of thermal management, even if the upfront cost is higher.
Powering The Future
The demand for better AI shows no signs of slowing down. As new capabilities arise, AI will unleash new value creation. The computing hardware for AI workloads will continue to require increasing amounts of power that will generate more heat.
Key technologies in the semiconductor industry, including chip packaging, power distribution and thermal management, will only grow in importance and drive new innovations to keep pace.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?