How did Google Gemini summarize a user's private document?

Google Gemini is a multimodal AI service that's built into Google products, like Gmail, Docs, and Sheets and works to do everything from summarize info to create pitch materials and generate images on the fly. 

Theoretically, you have to give Gemini and Google permission in order for it to read your Google Workspace content. According to Google's data privacy policy, "your Google Workspace content (emails, documents, etc.) will not be shared with other users without permission. This commitment covers all of our Google Workspace products for personal and business use, including Gmail, Google Docs, Sheets, Slides, Drive, Calendar, Meet, and Chat."

But last week, a user on X said Google Gemini summarized some of his private documents.

Tweet may have been deleted

"I just pulled up my tax return in @Google Docs--and unbidden, Gemini summarized it. So...Gemini is automatically ingesting even the private docs I open in Google Docs? WTF, guys. I didn't ask for this. Now I have to go find new settings I was never told about to turn this crap off."

The user, Kevin Bankster, said this could have just been a glitch for him. A Google spokesperson told Mashable over email that you have to proactively enable Gemini in Google Workspace and, once you do, Google has access to that information.

"Our generative AI features are designed to give users choice and keep them in control of their data," a spokesperson said. "Using Gemini in Google Workspace requires a user to proactively enable it, and when they do their content is used in a privacy-preserving manner to generate useful responses to their prompts, but is not otherwise stored without permission."

It's not entirely clear what's going on here. Tom's Hardware argues that there could be a couple of reasons behind the result — AI could be "hallucinating (lying)," Google's internal systems could be malfunctioning, or it could be a third, more elusive reason. 

"For Bankston, the issue seems localized to Google Drive, and only happens after pressing the Gemini button on at least one document. The matching document type (in this case, PDF) will subsequently automatically trigger Google Gemini for all future files of the same type opened within Google Drive. He additionally theorizes that it may have been caused by him enabling Google Workspace Labs back in 2023, which could be overriding the intended Gemini AI settings," Tom's Hardware suggests.

No matter what caused the slipup, it's not a great sign for AI in general, since there are already some privacy issues related to how data is collected, stored, and used by the generative systems.