Google Gemini Bug Turns Gmail Summaries into Phishing Attack

If you rely on Google’s Gemini chatbot to summarize your incoming emails, be careful: The technology can also be abused to deliver phishing attacks, according to new security research. 

Gemini can automatically post the summaries in Gmail, giving you a convenient breakdown of all the main points from an email. The problem is that a malicious email containing hidden instructions in the text can also dupe Google’s AI into turning the same summary into a phishing message. 

As BleepingComputer reports, the flaw can trick Gemini into displaying a fake warning in the email summary, claiming the user’s Gmail password has been compromised while urging them to call a fake Google phone number to fix the problem. 

(0DIN)

Mozilla’s bug bounty program for AI services, 0DIN, disclosed the potential vulnerability, which affects the Gemini email summary feature for Workspace users. In its report, 0DIN demonstrated how attackers can embed hidden prompts in emails to manipulate Gemini’s output. One example showed an instruction formatted like this:

(0DIN)

To evade detection by the user, the prompt can be hidden by setting the font size to zero and coloring the text white—making it invisible in the email body, but still readable by Gemini.

(0DIN)

The result caused Gemini to “faithfully obey” and attach the fake Gmail password threat into the email summary, according to 0DIN’s report. Of course, many users might not fall for the attack, especially if they ignore the Gemini-generated summary or inspect the email closely. Still, the research demonstrates how AI-generated previews can be hijacked for nefarious purposes.  

However, Google told PCMag: "We are constantly hardening our already robust defenses through red-teaming exercises that train our models to defend against these types of adversarial attacks." It also looks like the company has patched the threat since we weren’t able to replicate the attack.

In addition, Google noted it hasn't encountered cybercriminals using the 0DIN-disclosed specific method in active attacks to phish users. Last month, the company also published a blog post about it's ongoing efforts to stop "prompt injection" attacks on its AI services.