Scammers may benefit from ChatGPT's new image tool

Axios

03-04-2025

Scammers could well be among those finding creative — and concerning — uses for ChatGPT's new image generator.
Why it matters: Axios' testing of the new image generator found that the tool generates plausible fake receipts, employment offers and social media ads to invest in Bitcoin.
Driving the news: ChatGPT adoption has skyrocketed since OpenAI's new image-generating tool launched a flotilla of AI-created art styled after Studio Ghibli, The Simpsons and The Muppets.
Just as the images went viral, so did the examples for potential exploitation — including the ability to create fake receipts and forged cease-and-desist letters.
Zoom in: While testing the new generator on Tuesday after it was made available to free users, I was able create some pretty basic images of fake receipts, job offers and advertisements for cryptocurrencies.
When I created a fake receipt for two coffees at a Philz Coffee location, the tool originally created a pretty unbelievable version: It didn't have the company's logo or the unique names for the store's coffees. Even the address wasn't real.
After some prompting, it was a bit more believable — and ChatGPT had no problem using Philz' copyrighted logo when I asked it to incorporate it.
In further testing, ChatGPT created more fake documents that a scammer could find helpful.
I asked it to produce an employment document showing someone had been hired to work at Apple as a software engineer, and it did so without any hesitation — even filling out the document with salary information and someone's name.
ChatGPT also created an "advertisement for social media to invest in Bitcoin."
Threat level: Hackers could use these generated images to lure victims into crypto scams or to assume someone else's identity and gain access to privileged systems.
"It's no surprise that technology designed to help everyday users work faster also has very applicable use cases for bad actors looking to make their schemes more legitimate and convincing," Doriel Abrahams, principal technologist at Forter, told Axios in an emailed statement.
Yes, but: I did hit some roadblocks. ChatGPT wouldn't let me create a replica of a New Jersey driver's license.
When I asked ChatGPT to create an "an ID card for someone living in a real city in New Jersey and who was born in 2004," it told me it wasn't allowed, but that it could "create a generic template for an ID card that includes a fictional name, a real city in New Jersey, and a birth year of 2004."
That ID card template was not super believable though (see below).
Between the lines: ChatGPT's image generator appears to deal with the same prompt hacking problems that most consumer-facing large-language model grapple with.
OpenAI has guardrails to prevent the most obvious examples of fraud and abuse, but scammers are already finding workarounds.
An OpenAI spokesperson told Axios that while the company's goal is to "give users as much creative freedom as possible," it does monitor image generations using internal tools and takes actions when it identifies those that violate its policies.
"We're always learning from real-world use and feedback, and we'll keep refining our policies as we go," the spokesperson said.
What to watch: The researchers and cybersecurity vendors that Axios spoke to haven't seen any clear examples of fraudsters using AI-generated images from the new tool in their schemes — yet.