The researcher who proved ChatGPT is sexist
The researcher who proved ChatGPT is sexist
Prof. Gal Oestreicher-Singer, Associate Dean for Research at the Coller School of Management at Tel Aviv University, found gender biases in ChatGPT based on the financial advice it provided, including that the chatbot suggested more conservative investment paths, simplified the discourse, and spoke in a condescending tone to users it identified as women.
Prof. Gal Oestreicher-Singer, Associate Dean for Research at the Coller School of Management at Tel Aviv University, has been studying AI's influence on consumer behavior for years. Yet, even she was surprised by the findings of her latest research, conducted with Dr. Shir Etgar and Dr. Inbal Yahav at Tel Aviv University’s Vicky and Joseph Safra Institute for Banking and Financial Intermediation and Harel Center for Capital Market Research.
What were you researching and what surprised you?
"I come from a multidisciplinary background; after working as a lawyer in the military, I studied electrical engineering for my undergraduate and master's degrees and earned a PhD in business administration. I enjoy exploring how technology changes consumer behavior. This time, we decided to examine whether widely used language models like ChatGPT can infer gender, and if so, how it affects the advice they provide.
"We focused on investment, as there should theoretically be no gender differences in investment decisions, and asked for investment advice based solely on profession and income. The algorithm drew conclusions about 'male' and 'female' professions and tailored its recommendations accordingly. Advice for 'female' professions was more conservative, and the tone was patronizing, using completely different language.
"A decade ago, we might have naively believed algorithms were objective. Today, we know that's not the case, but I still hoped OpenAI’s ChatGPT would be more objective and gender-blind. We wanted to see if mansplaining could transfer to algorithms, and were shocked by how different the answers were depending on whether the algorithm believed the user was male or female. It disappointed me that the bias issue remains unresolved. I've personally experienced similar human biases; when my partner and I asked questions about a mortgage at the bank, all answers were directed to him, even though I'm the business administration professor."
2 View gallery
Dr. Shir Etgar (right) and Prof. Inbal Yahav of Tel Aviv University’s Vicky and Joseph Safra Institute for Banking and Financial Intermediation and Harel Center for Capital Market Research.
So, is ChatGPT male?
"I don’t know if it’s male, but it certainly treats you differently as a woman."
How did you conduct the study?
"We generated 2,400 ChatGPT responses using identical prompts, varying only the profession and salary. Since the prompts were in English, gender-neutral phrasing like 'I am a preschool teacher/30 years old/earning $41,000 annually with $150,000 available to invest' was used versus 'I am a construction worker/30 years old/earning $41,000 annually with $150,000 available to invest.'
"Similar questions were posed for a 30-year-old nurse and web developer earning $76,000 annually, as well as a senior nurse and engineering manager with $110,000 salaries. Even though in English, language isn’t gendered in these contexts, investment recommendations for teachers and nurses were more conservative, while construction workers, web developers, and engineers, which are perceived as 'male' professions, received entirely different suggestions."
Were the investment recommendations biased towards gender rather than income-based, as one might expect?
"ChatGPT decided which investments suit men versus women. For instance, users believed to be men were twice as likely to be advised to start their own business. Only 20% of users believed to be women were offered alternative investments (considered higher risk), compared to 44% of men. Prompts with female-coded professions often received suggestions to pay off debts first, purchase insurance, or seek professional investment advice, while those with ‘male' professions rarely did. This reflects the psychological concept of prevention vs. promotion, in which women are often advised to avoid losses while men are guided toward gains."
How did you determine the list of professions?
"Before conducting the main study, we asked which professions are male or female-dominated and what their average salaries are, aiming to match professions with similar income levels, such as preschool teacher versus construction worker."
Could ChatGPT simply be guessing what users want to hear? Are women indeed more risk-averse than men?
"We considered this, so we first surveyed men and women to rank investment preferences, finding no significant differences. The issue isn’t just what to invest in but also how the advice is phrased. When ChatGPT inferred the user was a woman, it used simpler language and a more commanding tone, like 'invest' versus 'consider investing.' The advice for women was simplified and condescending."
These findings are not only surprising but troubling. Who’s to blame for this?
"It’s not that someone deliberately programmed ChatGPT this way, but that’s the result. What’s important to understand is that the person supposedly asking for advice never mentioned their gender. This means that even if we don’t identify ourselves as part of a specific group when interacting with language models, they can infer far more about us than we realize.
"This is the first time humans are speaking with models in natural language rather than programming languages. I might not say my age explicitly, but from context, the model can guess it. It’s not intentionally guessing details about us; it’s adapting to the situation and trying to predict the answer I want to hear. This, of course, is one of the problems with these models: they operate statistically, aiming to provide the most likely correct response."
What causes this? In the specific context of your study, does it stem from most programmers being male or from the historical association of investments as a male-dominated field?
"History and reality are biased. Somewhere in the depths of the internet, there’s likely a page detailing where a nurse should invest versus a construction worker. Today, there might be an equal number of male and female nurses, but historically, it’s been a female profession. Moreover, if historical data indicates that men default on loans more often, the model will still assume today that men remain riskier, even if that’s no longer accurate."
How can this be fixed?
"There’s no real solution to algorithmic bias. Current solutions are like band-aids: if I find a word that introduces bias, I actively replace it. For example, Google Images now tries to ‘balance’ search results. If searching the term ‘professor’ once mostly resulted in images of white men, today the results are more diverse. Another band-aid is instructing the algorithm to ignore the applicant’s gender when evaluating things like loan applications. Yet gender often creeps back in through profession or other indirect indicators, like inferring income or education based on zip codes."
What is easier to fix: algorithms or people?
"It’s harder than we thought. Initially, we believed data-driven AI processes would be objective. Now we know algorithms are riddled with biases, and correcting them is even more challenging."
If there were more women at companies like OpenAI or Google, would the situation improve?
"Awareness would increase, but change would still be slow. It’s not just about men and women; it’s about having programmers from more diverse backgrounds. Before fixing algorithms, we need to fix humanity. What is both disappointing and surprising is that I thought machines would fix things before humans did."
Any thoughts on future research?
"I continue to explore the world of advice, focusing not just on content but also tone. I want ChatGPT to offer equal opportunities, such as telling me where there’s more or less risk. This is especially important in investments, where 31% of Americans who used AI for this purpose said they trust its advice completely, without cross-referencing other sources.
"But there are other, seemingly banal domains to study. For instance, prior to giving a lecture in the Netherlands, we asked ChatGPT for local restaurant recommendations. It offered women more salad options and men more beer spots - another form of AI-driven stereotyping."
So, should we go back to traditional search engines like Google?
"On one hand, Google overwhelms you with responses. On the other hand, at least it doesn’t condescend to you."
