Moving AI compute to phones massively reduces power use, study finds

Axios

8 hours ago

One of the easiest ways to minimize AI's environmental impact may be to move where the processing is done, per new academic research conducted in partnership with Qualcomm.
Why it matters: Running AI on devices instead of in the cloud slashes power consumption of queries by about 90%, the study finds.
The big picture: The AI boom is creating huge demands for power. It's a sufficiently important issue that the leaders of both the AI and energy industries are holding a series of high-level meetings this summer to sort out their mutual future.
Between the lines: That boom comes at an environmental cost. One oft-cited rule of thumb says querying an AI model consumes about 10 times the power of a Google search.
The industry has long touted the benefits of running models locally on devices instead of in the cloud — not just in energy terms, but also potentially making them cheaper and more private.
How it works: Researchers at the University of California, Riverside ran a series of experiments comparing the performance of various generative AI models, both in the cloud and on phones powered with Qualcomm chips.
Running any of six different models on the phones consumed anywhere from 75% to 95% less power, with associated sharp decreases in water consumption and overall carbon footprint.
The intrigue: Qualcomm is also developing an AI simulator and calculator that illustrates, for any given query and user location, what the responses would look like on-device versus the cloud, and how much less power and water they would use.
One example — running a coding skills question on the Llama-2-7B model in California — was 94% more power efficient and 96% more water efficient on-device.
What to watch: For all six models in the study, the inference time on the phones, measured in seconds, was higher than in the cloud.