4 days ago
OpenAI gets caught vibe graphing
During its big GPT-5 livestream on Thursday, OpenAI showed off a few charts that made the model seem quite impressive — but if you look closely, some graphs were a little bit off.
In one, ironically showing how well GPT-5 does in 'deception evals across models,' the scale is all over the place. For 'coding deception,' for example, GPT-5 apparently gets a 50.0 percent deception rate, but that's compared to OpenAI's smaller 47.4 percent o3 score which somehow has a larger bar.
Or this one, where one of GPT-5's scores is lower than o3's but is shown with a bigger bar. In this same chart, o3 and GPT-4o's scores are different but shown with equally-sized bars. That chart was bad enough that CEO Sam Altman commented on it, calling it a 'mega chart screwup.' An OpenAI marketing staffer also apologized for the 'unintentional chart crime.'
OpenAI didn't immediately respond to a request for comment. And while it's unclear if OpenAI used GPT-5 to actually make the charts, it's still not a great look for the company on its big launch day — especially when it is touting the 'significant advances in reducing hallucinations' with its new model.
Posts from this author will be added to your daily email digest and your homepage feed.
See All by Jay Peters
Posts from this topic will be added to your daily email digest and your homepage feed.
See All AI
Posts from this topic will be added to your daily email digest and your homepage feed.
See All News
Posts from this topic will be added to your daily email digest and your homepage feed.
See All OpenAI
