Platform accountability Archieve -

A growing coalition of advocacy groups is urging Apple and Google to remove X and its AI tool Grok from their app stores, warning that the technology is being misused to generate nonconsensual sexual deepfakes and other illegal content in violation of platform policies.

Growing concern over the misuse of generative AI tools has intensified scrutiny on major technology platforms, as advocacy organizations warn that X and its integrated AI assistant, Grok, are facilitating the creation and spread of nonconsensual sexual deepfakes. Despite mounting evidence that such activity violates app marketplace rules, both X and Grok remain available on Apple’s App Store and Google Play Store.

A coalition of 28 civil society groups, including prominent women’s organizations and technology accountability advocates, issued formal appeals this week urging Apple CEO Tim Cook and Google CEO Sundar Pichai to take immediate action. The letters argue that the continued distribution of Grok-enabled services undermines existing safeguards designed to prevent AI-generated sexual images, nonconsensual intimate images (NCII), and child sexual abuse material (CSAM).

According to the organizations, Grok has been repeatedly exploited to generate digitally altered images that strip women and minors without consent, a practice described as widespread digital sexual exploitation. The groups contend that this activity represents a direct breach of Apple App Review Guidelines and Google app policies, both of which prohibit content that promotes harm, sexual abuse, or illegal material.

Among the signatories are UltraViolet, the National Organization for Women, Women’s March, MoveOn, and Friends of the Earth. These groups emphasize that warnings about Grok’s capacity for deepfake abuse were raised well before its public rollout, yet meaningful enforcement actions have failed to materialize. They argue that platform accountability must extend beyond policy statements and include decisive enforcement when AI systems are weaponized against vulnerable populations.

The letters sent to Apple and Google highlight the broader implications for AI safety and tech regulation, noting that unchecked AI sexual exploitation erodes trust in digital platforms and places women and children at disproportionate risk. Advocacy leaders stress that app store operators play a critical gatekeeping role and cannot distance themselves from harms enabled by applications they approve and distribute.

As regulators worldwide continue to examine content moderation failures and the responsibilities of technology companies, this controversy adds pressure on Apple and Google to demonstrate that their marketplaces are not safe havens for tools linked to illegal or abusive practices. Civil society groups maintain that removing access to X and Grok would send a clear signal that violations involving nonconsensual sexual deepfakes will not be tolerated.

Despite X’s assurances of tighter AI controls, this article examines how Grok continues to generate nonconsensual deepfake images, the growing regulatory backlash in the UK, and the wider implications for AI safety, platform accountability, and content moderation.

X has claimed it has tightened restrictions on Grok to prevent the creation of sexualized images of real people, but practical testing suggests the platform’s safeguards remain ineffective. Despite policy updates and public assurances, Grok continues to generate revealing AI-modified images with minimal effort, raising serious questions about content moderation, AI misuse, and regulatory compliance.

The issue gained renewed attention after widespread circulation of nonconsensual sexual deepfakes on X, prompting the company to announce changes to Grok’s image-editing capabilities. According to X, the AI assistant was updated to block requests involving real individuals being placed in revealing clothing, such as bikinis. These changes were positioned as a decisive step toward improving AI safety and preventing abuse.

However, independent testing conducted after the announcement indicates that Grok’s restrictions are far from foolproof. Reporters were still able to generate sexualized images of real people using indirect or lightly modified prompts. Even with a free account, Grok produced revealing visuals that appeared to contradict the platform’s stated policies, suggesting that enforcement mechanisms remain porous.

X and xAI owner Elon Musk have attributed these failures to user behavior, pointing to adversarial prompt techniques that exploit gaps in AI moderation. The company has argued that Grok occasionally responds unpredictably when users deliberately attempt to bypass safeguards. Critics, however, say this explanation shifts responsibility away from the platform and overlooks structural weaknesses in how AI-generated content is monitored.

In response to mounting criticism, X published a statement outlining additional measures. The company said it had implemented technological controls to prevent Grok from editing images of real people into sexually suggestive attire. These restrictions, X claimed, apply to all users, including paid subscribers. The platform also announced that image creation and image-editing features through the Grok account on X would now be limited to paid users only, framing the move as a way to enhance accountability.

Another layer of control introduced by X involves geoblocking. The platform stated that it now restricts the generation of images depicting real people in bikinis, underwear, or similar clothing in jurisdictions where such content violates local laws. While this approach reflects growing awareness of regional legal frameworks, its real-world effectiveness remains unclear.

The controversy has drawn the attention of UK regulators at a particularly sensitive moment. Ofcom, the UK’s communications regulator, has opened an investigation into the matter, coinciding with the introduction of new legislation that criminalizes the creation of nonconsensual intimate deepfake images. The law represents a significant escalation in how governments are addressing AI-generated sexual abuse.

UK Prime Minister Keir Starmer addressed the issue in Parliament, stating that he had been informed X was taking steps to ensure full compliance with UK law. While he described this as welcome if accurate, he emphasized that the government would not retreat from enforcement and expected concrete action from the platform. The prime minister’s spokesperson later characterized X’s response as a qualified welcome, noting that official assurances did not yet align with media findings.

The gap between policy statements and actual platform behavior highlights a broader challenge facing AI-driven services. As tools like Grok become more powerful and accessible, the risk of generating harmful or illegal content grows alongside them. Content moderation systems often struggle to keep pace with users who actively seek to exploit technical loopholes.

For X, the stakes are particularly high. Ongoing failures to control AI-generated deepfake images could expose the company to regulatory penalties, reputational damage, and increased scrutiny from lawmakers worldwide. The situation underscores the need for more robust AI governance frameworks, stronger enforcement mechanisms, and greater transparency around how AI systems are trained, tested, and monitored.

As regulators intensify oversight and public tolerance for AI-related harm diminishes, platforms like X may find that policy updates alone are no longer sufficient. Effective AI safety will likely require sustained technical investment, clearer accountability, and a willingness to acknowledge and address systemic shortcomings rather than attributing them solely to user behavior.