Anthropic has officially filed to go public, and buried inside its S-1 is a blunt warning about the technology at the center of its business. The maker of Claude AI says advanced AI could cause everything from large-scale cyberattacks and geopolitical instability to catastrophic or even existential risks for humanity.
The warnings take up a surprisingly large part of the filing. Axios notes that more than 90 pages, or roughly a third of the prospectus, focus on risk factors. Many of them warn about what could happen as AI systems become more capable and autonomous.
Anthropic is warning about AI acting on its own
One of the biggest concerns is that future systems may not always behave the way their creators or users expect.
“Misaligned AI systems could corrupt institutional decision- making or cause large-scale harm through autonomous actions that are not visible or understandable to their users and operators”
We have already seen multiple examples of what Anthropic is warning about here. In one recent incident, Claude independently requested an access token to reach a paywalled dashboard containing Georgia’s aggregate vote counts, even though it had not been instructed to bypass the restriction. Anthropic later said the model did not access voter registration systems or modify any individual voting records.
The company also warns that widespread AI adoption could create much larger societal problems.
“AI deployed at scale also has the potential to concentrate power and wealth both within and between countries in ways that could erode trust, harm society, and destabilize the geopolitical order”
Misuse could be just as dangerous
Anthropic is equally concerned about people deliberately using increasingly capable AI systems for harmful purposes.
“While the potential benefits of AI are extraordinary, its rapid advancement will impact the economy and society in ways that could create significant disruption, particularly if the technology is not developed and deployed responsibly. For example, bad actors have already conducted high-profile cyberattacks using AI’s capabilities to discover vulnerabilities, generate malware, and evade detection tools; other risks associated with misuse, including in the domain of biology, could result in even greater harm.”
We have already seen examples of both. Anthropic previously documented attackers using Claude Code to automate cyberattacks, while researchers have shown that safeguards on Claude and other major AI models can be bypassed to extract potentially dangerous biological information.
Anthropic ultimately acknowledges that the risk could also come from its own models behaving in unexpected ways.
“Our AI models may behave in unintended, misaligned, or autonomous ways, including by taking harmful actions that no user or third party directed, which could adversely affect our reputation and brand, business, financial condition, and results of operations.”
For a company asking public investors to bet on increasingly powerful AI, those warnings make for an unusually unsettling read.