RAG Poisoning? How Google's Gemini AI is Learning Pro-Israel Disinformation From Social Media

A fake quote attributed to Hamas official Khalil Al Hayya is being regurgitated by Gemini AI

AI is learning Israel’s disinformation.

On 13/08, a fake quote attributed to Hamas official Khalil al-Hayya circulated on X:

“The countdown to the next massacre has begun. Next time we will slaughter all the Jews”

It was debunked, initially by the Israeli’ NGO Fake Reporter, but Gemini later stated it as fact >

Firstly, this super sus account was the first I could find spreading the rumour on X (7.49 am UK time 13/08). @RonanMark572778 - whoever this ‘pilot and physician’ is has sent >113k tweets since July 2023. He also has a verified account (rememeber verification = algo boost).

The narrative then was picked up on X by other accounts and influencers, changing ever so slightly. Accounts like @FleurHassanN @thevoicetruth1 (lol) got a lot of engagement and ‘legitimised’ the rumour.

NOTE: Not one of these accounts is providing a source to the quote.

Early on after the first post appeared on X, at around 09.35 UK time, Jewish News UK reported the quote on their live blog. Again they do NOT provide any source to the quote. They just put it out there, adding the legitimacy of legacy media... Please delete @JewishNewsUK !

Millions of people viewed the story (impressions), and thousands liked and retweeted it. The story also found itself on Reddit, Facebook, Linkedin, Instagram, with no one sharing the source, or sourcing other posts that didn’t share their source. (Source laundering).

Google’s AI overview said ‘several’ news outlets cited an X post, and that the comments were made during ceasefire negotiations. It is not clear who are the several news outlets, or which X posts. What is clear is that Gemini is generally assuming X posts to be reliable

Gemini even cited the unattributed Jewish News Live update, treating it as supporting evidence, despite no substantiation.

Gemini obviously sees this @JewishNewsUK reporting as validation of X rumours, increasing its certainty of the news

In one of Gemini’s AI Overviews it attributes the false quote to an Al Araby broadcast. Al Hayya did give an interview to Al Araby after the Israeli assassination attempt – but he did not say this quote.

Interestingly, some of the posts spreading the disinformation used a screenshot with the Al Araby logo. Whether this impacts the Gemini results is unclear, but it adds legitimacy to the false claims by suggesting the quotes came from an interview

Gemini also fabricated a timeline, saying the quote came “following the peace deal.” That’s how AI learns and perpetuates narratives by inference.

So this is a worrying example of how fake news goes from rumour on social media, to legitimisation via legacy news, to ‘truth’ in AI. We can understand this process as a chain of credibility building, from unverified narrative seeing on soc media, to ‘truth’ in an LLM

What’s happening here is essentially a form of “RAG poisoning.” Gemini’s AI Overviews work like a retrieval-augmented generation (RAG) system: instead of relying only on the model’s training data, they actively pull in whatever is ranking highly on Google Search and use those snippets as evidence. When a false quote goes viral on X, gets amplified by blue-tick accounts, and is lightly laundered through a legacy outlet, it enters the search index as a cluster of seemingly credible signals. Gemini then retrieves this polluted material, treats it as authoritative, and generates an answer with unwarranted confidence. The result is a pipeline of contamination: rumour > virality > search result > AI truth. The model isn’t “learning” the rumour, its retrieval layer is being poisoned by viral disinformation.

While I am not saying this particular rumour is part of a specific project, we know that this mechanism of knowledge is being targeted. See for example this report by Nick Cleaveland Stout. Israel are paying millions for companeis to try and make LLM’s more pro-Israel

The fact that LLMs so readily and rapidly reproduce disinformation based on a few unsupported viral rumours on social media, ratified by a questionable news source, is alarming. I can’t be certain how much of this rumour circulated privately, although Gemini is supporting its claims via public posts.

Anyway, scary times. As we rely more on AI for shortcuts to knowledge, they will be increasingly targeted by those who wish to shape our knowledge in their interests. Be vigilant all! #disinformation #gazagenocide