Apple has removed several AI-powered “nudify” apps from the App Store after coming under legal pressure from San Francisco’s city attorney, but the episode raises a much bigger question than whether a handful of apps violated App Store rules.
The real issue is why these apps made it onto one of the world’s most tightly controlled app marketplaces in the first place.
The latest controversy began after San Francisco City Attorney David Chiu sent cease-and-desist letters to both Apple and Google, accusing the companies of facilitating the distribution of apps capable of generating non-consensual AI-generated nude images. According to the legal notices, the companies have also profited from these apps through in-app purchases, with California authorities arguing that the platforms should sever ties with developers offering such services.
Apple has now responded, confirming that several of the identified apps have already been removed while additional developers risk permanent removal if they fail to comply with App Store policies.
Apple says the apps violate its rules. Critics say they shouldn’t have appeared at all.
Following the publication of WIRED’s investigation, Apple told 9to5Mac that apps designed to generate or distribute pornographic or “nudification” content have always violated its App Review Guidelines. According to the company, three of the apps highlighted by the San Francisco City Attorney’s Office have already been removed from the App Store, while the developer accounts behind them are in the process of being terminated. Apple also said it has contacted four additional developers, warning them to address policy violations or face removal from the platform.
“The App Store was designed to be a safe and trusted place for users,” Apple said in its statement, adding that it proactively rejects nudification apps and removes others after receiving reports through its reporting tools. Apple also reiterated that developers remain responsible for the content generated inside their applications. Apps that allow user-generated content must include mechanisms to filter objectionable material or risk removal from the App Store altogether.
Google has taken similar action. The company says it has already suspended hundreds of Android apps offering nudification features and restricted Play Store searches for terms like “nudify.” According to Google, the five Android apps cited by the San Francisco City Attorney have also been removed for violating Play Store policies.
Here’s Apple’s full statement as provided to 9to5Mac: “The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nudification’ apps are against our App Review Guidelines, and we have proactively rejected many of these apps and removed many others, including when users have flagged them via our reporting tools. We have removed three of the apps in question and are in the process of terminating their developer accounts from our program. We are in contact with four others that need to address policy violations or risk being removed as well.”
The bigger challenge is moderation in the AI era
The legal action comes amid growing concern over how generative AI has dramatically lowered the barrier to creating explicit deepfakes. Unlike traditional pornography, many of today’s AI-powered face-swapping tools require nothing more than a single photograph. Within seconds, users can generate fabricated nude images or explicit videos depicting real people without their knowledge or consent.
Authorities say the consequences extend well beyond privacy violations. Victims have increasingly reported harassment, extortion, reputational damage and severe mental health impacts. Previous investigations have also uncovered incidents across dozens of schools where AI-generated sexual images of minors were created using commercially available tools. What makes enforcement especially difficult is that many of these applications don’t openly advertise themselves as “nudify” tools. Instead, they present themselves as harmless AI face-swapping or photo-editing apps. The explicit image generation capabilities often remain hidden until after installation or behind in-app purchases, making automated moderation significantly harder.
Research published earlier this year found that a large number of seemingly ordinary face-swapping apps could still generate explicit deepfakes despite not marketing themselves that way. In many cases, researchers found little or no safeguards preventing misuse.
That distinction has become central to the debate now facing Apple and Google
Both companies maintain that their policies clearly prohibit pornography and sexually exploitative content. Critics, however, argue that policies alone are no longer enough when AI applications can disguise harmful functionality behind otherwise legitimate features. The City Attorney’s Office also alleges that Apple and Google financially benefit from these apps through commissions on in-app purchases, arguing that platform operators have a greater responsibility to identify and remove them before they reach users.
For Apple, the latest removals may help address the immediate criticism, but they are unlikely to end the broader conversation. As generative AI becomes cheaper, faster and increasingly capable of producing realistic synthetic media, app review is no longer just about checking screenshots or developer descriptions. It now requires identifying what an AI model can be persuaded to generate after installation, a considerably more difficult task than traditional moderation.
That challenge is unlikely to disappear anytime soon, and neither are the questions being asked of the companies running the world’s largest app stores.