Barely a few months ago, Wall Street’s big bet on generative AI had a moment of reckoning when DeepSeek arrived on the scene. Despite its heavily censored nature, the open source DeepSeek proved that a frontier reasoning AI model doesn’t necessarily require billions of dollars and can be pulled off on modest resources.
It quickly found commercial adoption by giants such as Huawei, Oppo, and Vivo, while the likes of Microsoft, Alibaba, and Tencent quickly gave it a spot on their platforms. Now, the buzzy Chinese company’s next target is self-improving AI models that use a looping judge-reward approach to improve themselves.
In a pre-print paper (via Bloomberg), researchers at DeepSeek and China’s Tsinghua University describe a new approach that could make AI models more intelligent and efficient in a self-improving fashion. The underlying tech is called self-principled critique tuning (SPCT), and the approach is technically known as generative reward modeling (GRM).
In the simplest of terms, it is somewhat like creating a feedback loop in real-time. An AI model is fundamentally improved by scaling up the model’s size during training. That takes a lot of human work and computing resources. DeepSeek is proposing a system where the underlying “judge” comes with its own set of critiques and principles for an AI model as it prepares an answer to user queries.
This set of critiques and principles is then compared against the static rules set at the heart of an AI model and the desired outcome. If there is a high degree of match, a reward signal is generated, which effectively guides the AI to perform even better in the next cycle.
The experts behind the paper are referring to the next generation of self-improving AI models as DeepSeek-GRM. Benchmarks listed in the paper suggest that these models perform better than Google’s Gemini, Meta’s Llama, and OpenAI’s GPT-4o models. DeepSeek says these next-gen AI models will be released via the open-source channel.
Self-improving AI?
The topic of AI that can improve itself has drawn some ambitious and controversial remarks. Former Google CEO, Eric Schmidt, argued that we might need a kill switch for such systems. “When the system can self-improve, we need to seriously think about unplugging it,” Schmidt was quoted as saying by Fortune.
The concept of a recursively self-improving AI is not exactly a novel concept. The idea of an ultra-intelligent machine, which is subsequently capable of making even better machines, actually traces all the way back to mathematician I.J. Good back in 1965. In 2007, AI expert Eliezer Yudkowsky hypothesized about Seed AI, an AI “designed for self-understanding, self-modification, and recursive self-improvement.”
In 2024, Japan’s Sakana AI detailed the concept of an “AI Scientist” about a system capable of passing the whole pipeline of a research paper from beginning to end. In a research paper published in March this year, Meta’s experts revealed self-rewarding language models where the AI itself acts as a judge to provide rewards during training.
Meta’s internal tests on its Llama 2 AI model using the novel self-rewarding technique saw it outperform rivals such as Anthropic’s Claude 2, Google’s Gemini Pro, and OpenAI’s GPT-4 models. Amazon-backed Anthropic detailed what they called reward-tampering, an unexpected process “where a model directly modifies its own reward mechanism.”
Google is not too far behind on the idea. In a study published in the Nature journal earlier this month, experts at Google DeepMind showcased an AI algorithm called Dreamer that can self-improve, using the Minecraft game as an exercise example.
Experts at IBM are working on their own approach called deductive closure training, where an AI model uses its own responses and evaluates them against the training data to improve itself. The whole premise, however, isn’t all sunshine and rainbows.
Research suggests that when AI models try to train themselves on self-generated synthetic data, it leads to defects colloquially known as “model collapse.” It would be interesting to see just how DeepSeek executes the idea, and whether it can do it in a more frugal fashion than its rivals from the West.