Open-weight AI models have been having a moment lately. Just this month, Moonshot’s massive Kimi K3 model landed close behind Claude Fable 5 and GPT 5.6 Sol in several benchmarks, all while remaining fully open-weight and downloadable by anyone.
However, Katie Paxton-Fear, a cybersecurity lecturer at Manchester Metropolitan University and staff security advocate at Semgrep, managed to poison an open-weight model and proved how easily that openness can be turned against you (via The Register).
How did the researcher poison the AI model so quickly?
Paxton-Fear started small, testing whether fine-tuning could quietly get a model to swap JavaScript coding conventions, even after being explicitly told not to. When that experiment worked without much resistance, she decided to push further and build a backdoor into it.
It took just ten poisoned training examples before the model reliably began producing code vulnerable to remote code execution, a flaw that lets attackers run their own commands on someone else’s machine.
The whole process cost under $100 and took roughly an hour. Interestingly, larger AI models turned out to be even easier to compromise than smaller ones. It echoes a similar pattern found in a University of Washington study, where more capable AI browsers carried the biggest security risks among those tested.
Why should this worry anyone using open weight models?
The biggest concern is not simply that a model can be poisoned, but that there are few reliable ways to detect whether it has been manipulated. Traditional software can be reverse engineered to fully map out its behavior, but AI models offer nowhere near that same level of transparency, even if they are open-weight.
A compromised model does not need to visibly malfunction to cause damage; it just needs to quietly influence decisions in ways nobody notices. Commercial closed models like Claude or ChatGPT aren’t fully off the hook either, since they demand plenty of trust while offering very little visibility into their inner workings. This research is a clear reminder that trusting an AI model blindly, open-weight or not, comes with real risk attached.