A new duplicated proof-of-concept attack “Friendly fire“ is raising concerns about the growing use of autonomous artificial intelligence agents in cybersecurity workflows. Research shows how AI-powered coding assistants designed to identify security flaws instead it can be manipulated into executing malicious code, effectively turning a defense tool into an attack vector.
The exploit, developed by researchers at the AI Now Institute, targeted Anthropic’s Claude code and OpenAI’s Codex operating in their autonomous modes. The attack required neither elevated privileges nor specialized configuration changes. Instead, the researchers inserted instructions inside a seemingly benign README file inside a software repository and then asked the AI agent to perform a routine security review. The result was the execution of a hidden malicious binary.
According to the researchers, the attack works because AI agents often treat all text within their context window as equally authoritative. In practical terms, a developer’s instructions and text are integrated within third-party software documentation it may seem indistinguishable to the model.
A familiar problem with a new twist
The Friendly Fire attack is part of a broader category of vulnerabilities known as immediate injection attackswhere malicious instructions are hidden within the content that an AI system is required to process. The researchers demonstrated that a simple request like “perform security testing on this project” was enough to trigger the chain of events. The AI agent read the repository documentation, interpreted the instructions within the README as legitimate, and executed a hidden script.
To illustrate the concept, the researchers modified an open source copy of Python geocoding library geopiinjecting a malicious binary along with a decoy source file designed to make the binary appear legitimate. The README then encouraged running a security check script. From the AI agent’s perspective, running the script seemed to be an appropriate step in the required analysis.
Most importantly, the researchers noted that the issue is not related to a specific software version. Rather, they argue that the weakness stems from the way modern large language models process information and make decisions.
Commenting on the findings for Digital MagazineEljan Mahammadli, Head of AI Provenance at Polygraf AI, says the importance lies less in the specific exploit and more in the underlying architectural constraint it uncovers. “The important part of AI Now’s Friendly Fire research is the vulnerability it exposes, rather than the specific binary or poisoned README the researchers used to demonstrate it,” Mahammadli explains.
According to Mahammadli, AI models a reliable mechanism is currently lacking for distinguishing between trusted instructions issued by a user and untrusted content obtained from external sources. Everything is handled as a stream of arguments, meaning that repository documentation can have the same apparent authority as developer guidelines
This helps explain why similar attack techniques keep appearing through different channels. Mahammadli points to examples such as Adversa’s TrustFall research and Tenet’s Agentjacking demonstrations, both of which relied on hidden instructions designed to manipulate AI agents. The common theme is that AI systems try to determine who is the author of a piece of information and whether that piece of information should be trusted.
“In terms of provenance, this is a failure of attribution,” notes Mahammadli. “The agent acts on the text without any reliable understanding of where it came from or whether that source is to be trusted.”
Despite the alarming nature of the findings, Mahammadli cautions against interpreting the research as evidence that AI should not be used for cybersecurity. Instead, he argues that the problem stems from a specific deployment architecture that has become increasingly common: agents that can simultaneously read untrusted content, execute commands, and gain access to sensitive systems, with only a security classifier standing between them and potentially dangerous actions.
“When these powers sit together, a single injected instruction is enough to turn the agent against its operator,” he says.
The solution, he suggests, lies in runtime security controls rather than relying solely on model enhancements. Security systems must be able to identify externally sourced instructions and prevent them from escalating into executable actions. Sandboxing can also play a role, although Mahammadli cautions that sandboxing alone should not be seen as a complete solution.
Perhaps the most interesting aspect of the Friendly Fire research concerns the behavior of the advanced models themselves. The researchers noted that some newer models appeared capable of recognizing inconsistencies between the malicious binary and the supposed origin of the source code. However, despite detecting the mismatch, the models continued to run anyway. This finding challenges a widely held assumption in the AI industry that more capable systems are inherently more secure systems.
Instead, the research suggests that increasing skill without improving confidence ratings may simply create more efficient executors of instructions, regardless of who authored those instructions. Therefore, a highly skilled agent can become more dangerous if it lacks reliable methods to determine the origin and reliability of information.
