AI's elusive coding speedup

Axios

8 hours ago

A surprising new study finding that AI tools can reduce programmers' productivity is upending assumptions about the technology's world-changing potential.
Why it matters: Software runs our civilization, and AI is already transforming the business of making it — but no one really knows whether AI will decimate programming jobs, or turn every coder into a miracle worker, or both.
Driving the news: The study by METR, a nonprofit independent research outfit, looked at experienced programmers working on large, established open-source projects.
It found that these developers believed that using AI tools helped them perform 20% faster — but they actually worked 19% slower.
The study appears rigorous and well-designed, but it's small (only 16 programmers participated, completing 246 tasks).
Zoom out: For decades, industry visionaries have dreamed of a holy grail called "natural language programming" that would allow people to instruct computers using everyday speech, without needing to write code.
As large language models' coding prowess became evident, it appeared this milestone had been achieved.
"The hottest new programming language is English," declared AI guru (and OpenAI cofounder) Andrej Karpathy on X early in 2023, soon after ChatGPT's launch.
In February, Karpathy also coined the term "vibe-coding" — meaning the quick creation of rough-code prototypes for new projects by just telling your favorite AI to whip up something from scratch.
The most fervent believers in software's AI-written future say that human beings will do less and less programming, and engineers will turn into some combination of project manager, specifications-refiner and quality-checker.
Either that, or they'll be unemployed.
Zoom in: AI-driven coding tends to be more valuable in building new systems from the ground up than in extending or refining existing systems, particularly when they're big.
While innovative new products get the biggest buzz and make the largest fortunes, the bulk of software work in most industries consists of more mundane maintenance labor.
Anything that makes such work more efficient could save enormous amounts of time and money.
Yes, but: This is where the METR study found AI actually slowed experienced programmers down.
One key factor was that human developers found AI-generated code unreliable and ended up devoting extra time to reviewing, testing and fixing it.
"One developer notes that he 'wasted at least an hour first trying to [solve a specific issue] with AI' before eventually reverting all code changes and just implementing it without AI assistance," the study says.
Between the lines: The study authors note that AI coding tools are improving at a rapid enough rate that their findings could soon be obsolete.
They also warn against generalizing too broadly from their findings and note the many counter-examples of organizations and projects that have made productivity gains with coding tools.
One notable caution that's inescapable from the study's findings: Don't trust self-reporting of productivity outcomes. We're not always the best judges of our own efficiency.
Another is that it's relatively easy to measure productivity in terms of "task completion" but very hard to assess total added value in software-making.
Thousands of completed tickets can be meaningless — if, for instance, a program is about to be discontinued. Meanwhile, one big new insight can change everything in ways no productivity metric can capture.
The big picture: The software community is divided over whether to view the advent of AI coding with excitement or dread.