The latest ChatGPT is supposed to be ‘PhD level' smart. It can't even label a map

Yahoo

2 days ago

A version of this story appeared in CNN Business' Nightcap newsletter. To get it in your inbox, sign up for free here.
Sam Altman, the artificial intelligence hype master, is in damage-control mode.
OpenAI's latest version of its vaunted ChatGPT bot was supposed to be 'PhD-level' smart. It was supposed to be the next great leap forward for a company that investors have poured billions of dollars into.
Instead, ChatGPT got a flatter, more terse personality that can't reliably answer basic questions. The resulting public mockery has forced the company to make sweaty apologies while standing by its highfalutin claims about the bot's capabilities.
In short: It's a dud.
The misstep on the model, called GPT-5, is notable for a couple of reasons.
1. It highlighted the many existing shortcomings of generative AI that critics were quick to seize on (more on that in a moment, because they were quite funny).
2. It raised serious doubts about OpenAI's ability to build and market consumer products that human beings are willing to pay for. That should be particularly concerning for investors, given OpenAI, which has never turned a profit, is reportedly worth $500 billion.
Let's rewind a bit to last Thursday, when OpenAI finally released GPT-5 to the world — about a year behind schedule, according to the Wall Street Journal. Now, one thing this industry is really good at is hype, and on that metric, CEO Sam Altman delivered.
During a livestream ahead of the launch last Thursday, Altman said talking to GPT-5 would be like talking to 'a legitimate PhD-level expert in anything, any area you need.'
In his typically lofty style, Altman said GPT-5 reminds him of 'when the iPhone went from those giant-pixel old ones to the retina display.' The new model, he said, is 'significantly better in obvious ways and subtle ways, and it feels like something I don't want to ever have to go back from,' Altman said in a press briefing.
Then people started actually using it.
Users had a field day testing GPT-5 and mocking its wildly incorrect answers.
The journalist Tim Burke said on Bluesky that he prompted GPT-5 to 'show me a diagram of the first 12 presidents of the United States with an image of their face and their name under the image.'
The bot returned an image of nine people instead, with rather creative spellings of America's early leaders, like 'Gearge Washingion' and 'William Henry Harrtson.'
A similar prompt for the last 12 presidents returned an image that included two separate versions of George W. Bush. No, not George H.W. Bush, and then Dubya. It had 'George H. Bush.' And then his son, twice. Except the second time, George Jr. looked like just some random guy.
Labeling basic maps of the United States also proved tricky for GPT-5 (but again, pretty funny, as tech writer Ed Zitron's post on Bluesky showed).
GPT-5 did slightly better when I asked it on Wednesday for a map of the US. Some people can, in fact, label the great state of Vermont correctly without a PhD, but not GPT-5. And this is the first I'm hearing of states named 'Yirginia.'
The slop coming out of GPT-5 was funny when it was just us nerds trying to find its blind spots. But some regular fans of ChatGPT weren't laughing. Especially because users have been particularly alarmed by the new version's personality – or rather, lack thereof.
In rolling out the new model, OpenAI essentially retired its earlier models, including the wildly popular GPT-4o that's been on the market for over a year, making it so that even people who loved the previous iteration of the chatbot suddenly couldn't use it. More than 4,000 people signed a Change.org petition to compel OpenAI to resurrect it.
'I'm so done with ChatGPT 5,' one user wrote on Reddit, explaining how they tried to use the new model to run 'a simple system' of tasks that an earlier ChatGPT model used to handle. The user said GPT-5 'went rogue,' deleting tasks and moving deadlines.
And while OpenAI's defenders could chalk that up to an isolated or even made-up incident, within 24 hours of the GPT-5 launch Altman was doing damage control, seemingly caught of guard by the bad reception. On X, he announced a laundry list of updates, including the return of GPT-4o for paid subscribers.
'We expected some bumpiness as we roll out so many things at once,' Altman said in a post. 'But it was a little more bumpy than we hoped for!'
The CEO's failure to anticipate the outrage suggests he doesn't have a firm grasp on how an estimated 700 million weekly active users are engaging with his product.
Perhaps Altman missed all the coverage — from CNN, the New York Times, the Wall Street Journal — of people forming deep emotional attachments to ChatGPT or rival chatbots, having endless conversations with them as if they were real people. A simple search of Reddit could have offered insights into how others are integrating the tool into their workflows and lives. Basic market research should have shown OpenAI that a mass update sunsetting the tools people rely on would be more than just a bit bumpy.
When asked about the backlash to GPT-5, an OpenAI representative pointed CNN to Altman's public statements on social media announcing the return of older models, as well as a blog post about how the company is optimizing GPT-5.
The messy rollout speaks to how the AI industry as a whole is struggling to prove themselves as producers of consumer goods rather than 'labs' — as they love to call themselves, because it sounds more scientific and distracts people from the fact that they are backed by people who are trying to make unfathomable amounts of cash for themselves.
AI companies often base their fanfare around how a model performs in various behind-the-scenes benchmark tests that show how well a bot can do complex math. For all we know, GPT-5 sailed through those evaluations.
But the problem is that OpenAI hyped the thing so far into the stratosphere, disappointment was (or should have been) inevitable.
'I honestly didn't think OpenAI would burn the brand name on something so mid,' wrote prominent researcher and AI critic Gary Marcus. 'In a rational world, their valuation would take a hit,' he added, noting OpenAI still hasn't turned a profit, is slashing prices to keep its user numbers up, and is hemorrhaging talent as competition heats up.
For critics like Marcus, the GPT-5 flop was a kind of vindication. As he noted in a blog post, other models like Elon Musk's Grok aren't faring much better, and the backlash from even AI proponents feels like a turning point.
When people talk about AI, they're talking about one of two things: the AI we have now — chatbots with limited, defined utility — and the AI that companies like Altman's claim they can build — machines that can outsmart humans and tell us how to cure cancer, fix global warming, drive our cars and grow our crops, all while entertaining and delighting us along the way.
But the gap between the promise and the reality of AI only seems to widen with every new model.
CNN's Lisa Eadicicco contributed reporting.