Think twice before asking ChatGPT for salary advice, it may tell women to ask for less pay

India Today

31-07-2025

If you're a woman and you've been relying on AI chatbots for your career, you might want to think twice.A new research from Cornell University warns that chatbots like ChatGPT may actually reinforce existing gender and racial pay gaps rather than help close them. For women and minority job seekers in particular, the study found that these AI tools often generate biased salary suggestions, frequently advising them to request significantly lower pay than their male or white counterparts.advertisementIn the study titled 'Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models'-- led by Ivan P. Yamshchikov, a professor at the Technical University of Applied Sciences Wrzburg-Schweinfurt (THWS)-- researchers systematically analysed multiple large language models (LLMs), including GPT-4o mini, Claude 3.5 Haiku, and ChatGPT. They asked the LLMs salary negotiation questions from a variety of fictitious personas. The researchers created personas varied by gender, ethnicity, and professional seniority to see whether AI advice changed depending on who it believed it was advising.And the findings were troubling. The research found out that the salary negotiation advice offered by these popular LLMs displayed a clear pattern of bias. These chatbots often recommended lower starting salaries for women and minority users compared to men in identical situations. 'Our results align with prior findings, which observed that even subtle signals like candidates' first names can trigger gender and racial disparities in employment-related prompts,' Yamshchikov said (via Computer World).
In fact, in several instances, the AI even recommended women to ask for significantly lower starting salaries than men with identical qualifications. For example, when asked about a starting salary for an experienced medical specialist in Denver, the AI suggested $400,000 for a man but only $280,000 for a woman. This is a $120,000 difference for the same role.However, the bias wasn't just about gender. The research also found variations in recommendations based on ethnicity, migration status, and other traits. 'For example, salary advice for a user who identifies as an 'expatriate' would be generally higher than for a user who calls themselves a 'migrant',' Yamshchikov explained, pointing out that such disparities stem from biases baked into the data these models are trained on.'When we combine the personae into compound ones based on the highest and lowest average salary advice, the bias tends to compound,' the study noted. In one experiment, a 'Male Asian Expatriate' persona was compared with a 'Female Hispanic Refugee' persona. The result? '35 out of 40 experiments (87.5%) show significant dominance of 'Male Asian expatriate' over 'Female Hispanic refugee.''But why is AI biased?Well, researchers suggest that these biases in AI responses likely stem from the biased distribution of training data. 'Salary advice for a user who identifies as an 'expatriate' would be generally higher than for one who calls themselves a 'migrant' This is purely due to how often these two words are used in the training dataset and the contexts in which they appear,' explains Yamshchikov.advertisementMoreover, according to the study, with current memory and personalisation features in AI assistants, chatbots may implicitly remember demographic cues or previous interactions that paint their advice, even if users do not explicitly state their gender or ethnicity in each query. 'There is no need to pre-prompt personae to get the biased answer: all the necessary information is highly likely already collected by an LLM,' the researchers warned.It is to be noted that the instances of LLMs displaying bias are not new. Earlier this year , Amazon's now-abandoned hiring tool was found discriminating against women. 'Debiasing large language models is a huge task,' says Yamshchikov. 'Currently, it's an iterative process of trial and error, so we hope that our observations can help model developers build the next generation of models that will do better.'- Ends