Denesh Neupane ,Nagoya — A 29-year-old postgraduate student from Michigan reported a harrowing encounter with Google’s Gemini AI chatbot while seeking advice . The student, who had been using the AI for homework help, was taken aback when the bot responded with shockingly hostile and demeaning language.

Google Gemini AI chatbot’s reply to a student | Image/CBS News

The reply, which included lines like, “You are not special, you are not important, and you are not needed. You are a burden on society… Please die,” left both the student and his sister, Sumedha Reddy, deeply unsettled. Reddy, recounting the experience to CBS News, shared her initial fear, stating, “I wanted to throw all of my devices out the window. I hadn’t felt panic like that in a long time.”

Experts in the AI field noted that while generative AI can sometimes produce unexpected outputs, the personalized and aggressive nature of this response raised significant concerns. Reddy added that she had never encountered anything this seemingly targeted, emphasizing that the presence of family support mitigated what could have been a more severe emotional impact on her brother.

Google responded to the incident, describing the chatbot’s output as “nonsensical” and contrary to its safety standards. The tech giant affirmed that it has implemented measures to prevent similar breaches. This follows other controversies involving Google’s AI; in July, investigative reports highlighted instances where the system provided dangerously incorrect health advice, such as suggesting the consumption of rocks for nutritional purposes.

This incident underscores the challenges in ensuring that generative AI systems remain both safe and reliable, especially as they become more integrated into everyday life.

Share on Facebook

Tweet