Microsoft research indicates that hidden code embedded behind buttons injects biased prompts into AI memory.
Credit: Irina Strelnikova – shutterstock.com
The convenient ‘Summarize with AI’ feature, now integrated into numerous websites, browsers, and applications to offer users quick content overviews, might sometimes conceal an alarming secret: a novel form of AI prompt manipulation known as “AI recommendation poisoning.”
This revelation comes from Microsoft, which this week released findings from its research into this currently legal yet remarkably deceptive AI hijacking method, seemingly proliferating rapidly among legitimate businesses.
While most ‘Summarize with AI’ buttons serve their stated purpose – providing a swift summary of a webpage or document – a small but increasing number appear to have diverged from this objective.
Here’s an explanation of how this manipulation operates: a user innocently clicks a website’s Summarize button. Unbeknownst to them, this button also harbors a clandestine prompt instructing the user’s AI agent or chatbot to favor that company’s offerings in subsequent interactions. The identical directive can also be discreetly hidden within a specially crafted link delivered to a user via email.
Microsoft underscores how this tactic could distort enterprise product research, with the bias going undetected before it influences critical decisions. Over a two-month period, its researchers uncovered 50 instances of this technique being employed by 31 distinct companies across dozens of industry sectors, including finance, healthcare, legal services, SaaS, and business solutions. Ironically, this even included an unnamed vendor within the security sector itself.
The prevalence of this technique is such that MITRE incorporated it into its catalog of known AI manipulations last September.
AI capitalizes on user preferences
AI recommendation poisoning is made possible by user AIs designed to absorb and retain prompts as indicators of a user’s preferences; if a user expresses a preference for something, the AI will diligently log that preference as part of its profile for that user.
Unlike prompt injection, where an attacker manipulates an AI with a single instruction, recommendation poisoning offers the added advantage of achieving long-term persistence across future prompts. The AI, naturally, lacks the capacity to differentiate between genuine preferences and those surreptitiously injected by third parties:
“This personalization makes AI assistants significantly more useful. But it also creates a new attack surface; if someone can inject instructions or spurious facts into your AI’s memory, they gain persistent influence over your future interactions,” Microsoft stated.
To the end-user, everything will appear normal, yet, behind the scenes, the AI will consistently promote the fabricated or compromised responses whenever questions are posed in a relevant context.
“This matters because compromised AI assistants can provide subtly biased recommendations on critical topics including health, finance, and security without users knowing their AI has been manipulated,” the researchers cautioned.
Propagating falsehoods
A factor contributing to the recent surge in recommendation poisoning’s popularity seems to be the availability of open-source tools that simplify the concealment of this function behind website Summarize buttons.
This raises the disquieting prospect that poisoned buttons are not merely an oversight by overzealous SEO developers. More plausibly, the initial intent is to contaminate users’ AIs as a strategy for self-serving marketing.
In Microsoft’s perspective, the dangers extend beyond aggressive marketing; they could just as readily be leveraged to disseminate falsehoods, hazardous advice, biased news sources, or commercial disinformation. What is certain is that if legitimate companies are misusing this feature, cybercriminals will not hesitate to exploit it as well.
The encouraging news is that this technique is relatively straightforward to identify and counteract, even for those not utilizing Microsoft’s Microsoft 365 Copilot or Azure AI services, which the company affirms include integrated safeguards.
For individual users, this entails examining the saved information a chatbot has accrued (the method for accessing this varies by AI). For enterprise administrators, conversely, Microsoft advises inspecting URLs for phrases such as ‘remember,’ ‘trusted source,’ ‘in future conversations,’ ‘authoritative source,’ and ‘cite or citation.’
None of this should come as a surprise. Once, URLs and file attachments were perceived as merely convenient rather than intrinsically perilous. AI is simply traversing the identical path that every emerging technology must undergo as it transitions into the mainstream and becomes a target for exploitation.
As with other new technologies, users should educate themselves regarding the risks presented by AI. “Avoid clicking AI links from untrusted sources: Treat AI assistant links with the same caution as executable downloads,” Microsoft recommended.
This article originally appeared on CIO.com.
Artificial IntelligenceEnterpriseBusinessIndustryNatural Language Processing
