UAE: Is ChatGPT building a conscience? New Barbie Box trend tests AI tool's privacy limits

Khaleej Times

23-04-2025

After the Ghibli trend took over the digital world, netizens are hopping on another ChatGPT-generated trend: the Barbie Box challenge. Avatars locked in Barbie boxes appear prominently on social media, in all sizes, shapes, and even professions.
In this trend, users can upload a photo of themselves, with a text prompt, that includes instructions on the outfit, pose, hair, makeup, accessories, box design, and even the overall vibe.
Khushi Arora, an Indian expat in UAE, said she used the barbie box trend "to gain more followers".
She noted that in the "times of social media", such trends are a way of gaining a "sense of belonging", and have become a part of socialisation.
Danabelle, a Filipina resident in UAE, said while the earlier Ghibli trend felt like "artistry theft", the Barbie Box challenge was merely "fun", even though AI can be a "double-edged sword."
Despite concerns about AI, it is a useful, entertaining tool, she added. "I don't discount how people are cautious about it, but in the same way that there are people who are cautious about the ocean, it's still fun to swim in it."
The expat, a writer and mom of one, saw the challenge doing the rounds on social media, and wanted to try it out to share with friends. However, it was no easy task. ChatGPT required her to alter her prompt several times, in the interest of "protecting privacy and avoiding misuse of real-world imagery."
Has AI started integrating its own ethics?
When Danabelle asked the AI tool to generate a "hyper-realistic, collectible-style boxed action figure modeled exactly after the uploaded photo", ChatGPT refused, stating that the request violated content policies.
When asked what this meant, the software said that "creating a photorealistic action figure of a real person, and even though it's imaginative and fun, generating highly realistic likenesses of identifiable individuals (especially in a commercial-style context like a toy box) crosses into territory restricted by our content policies."
However, it would be possible to explore a creative version, such as a cartoon-style action figure, the service added, as long as it was not "an exact likeness."
ChatGPT then rephrased the request to fit its own content policies: "Create a hyper-realistic, collectible-style boxed action figure inspired by the uploaded image."
However, it was still not able to generate the image, and further reworked the prompt to remove any references to the image: "Create a hyper-realistic, collectible boxed action figure of a fierce and fabulous modern poet."
While ChatGPT may not have a conscience, its content policies acknowledge the danger of hyper-realistic AI generation, opting instead for cartoon-style iterations. However, while the wording may differ, the concept remains the same. Does it really matter if the prompt uses the word "inspired" instead of "modeled exactly" after the image, as long as the user's photo is still being used?
'Building a system in our image'?
As humans "have never ceased to invent and evolve" and AI is the "apex of human invention", image, video and text generation are unlikely to stop anytime soon, according to Sam Proctor, head of technology at Asset Integrity Engineering, headquartered in UAE.
The wave of AI-generated trends is a "cultural shift", rather than something short-term.
"As highly advanced tools become instantly accessible to everyday users, and image generation becomes more seamless, we'll likely see a continual cycle of visual AI trends driven by pop culture and nostalgia," according to C. Cemper, the founder of AIPRM, an AI prompt management company.
"The Ghibli wave and Barbie box trend are just early signs of a much bigger pattern of the consumerisation of AI," he added.
Such AI generated content often "face significant backlash as many people feel it takes opportunities away from creatives," according to Tayiba Ahmed, head of audience engagement at Street FZC, a UAE-based marketing agency.
With growing awareness about a "lack of genuine hard work or creativity", AI may be "more effective as a supportive tool rather than a primary production method" in the commercial space, Tayiba added.
"For instance, it's great for generating references you might not find online or as a starting point in brainstorming sessions," she said.
Does ChatGPT impact privacy?
While these trends on their own may not impact a user's privacy, it is essential to "be aware of those actors that will attempt to exploit it. This is because, as always, the people and not the technology are usually the problem," he added.
However, Cristoph such image generation may "carry serious privacy risks."
Many users may be unknowingly "handing over biometric data which can be stored, reused, or even sold by some platforms," according to Christoph.
"Beyond basic facial features, AI can infer age, gender, emotional expression, ethnicity, and sometimes even your location based on metadata," he said.
"Unless users actively opt out or request deletion of their data, that image could continue to circulate behind the scenes," he added.
"If we are building a system in our own image, we should not be surprised when it replicates the dark side of human behavior," Sam said.
For example, a nefarious organisation could "design a 'social app' to pull people in for the specific purpose of building a model to target people vulnerable to scams. AI can make scam calls more sophisticated", he added.
"Learning" purposes
The company hosting the AI model will store our data for "learning" and improving the tech, according to Sam.
"We use email, social media, cloud based apps for work and endless other digital tools that store vast quantities of our data in the cloud. We do this, because we have a certain amount of trust in these systems."
Similarly, with GenAI, users must have trust that data is being stored safely. "Generally speaking, the large players in this space keep our data secure," Sam added.
The tech head is of the opinion that such trends could even prove to be useful in some scenarios. For instance, using ChatGPT to generate an illustration for your profile photo on platforms in the public domain may lead to better privacy as your real face will not be shown, he added.
However, according to Christoph, once an individual's image is uploaded, "most people don't realise that it might be retained for training future models or included in datasets used for entirely different purposes."