The rapid advancement of generative artificial intelligence (AI) has intensified challenges related to deepfakes, impersonation scams, and manipulated content across social media platforms. 

As synthetic media becomes easier to create and harder to detect, companies are being forced to adopt more sophisticated trust and safety strategies. 

In an email interview with eSecurityPlanet, Alexandra Ryabova, COO of Wizz App, discussed how identity verification, AI moderation, and regulations like the TAKE IT DOWN Act are strengthening user safety on social media. 

Key Takeaways

• Deepfakes and AI-generated scams are becoming more difficult to detect, enabling bad actors to create fake identities, voice clones, and manipulated content at scale.

• Identity verification is emerging as a key defense strategy, with some social media platforms using age verification and facial-matching technologies to reduce fake accounts.

• AI moderation helps platforms identify harmful content at scale, but human oversight remains critical to address contextual errors and false positives.

• The TAKE IT DOWN Act requires platforms to remove reported AI-generated non-consensual intimate imagery within 48 hours, increasing accountability for social media companies. 

The Growing Threat of Deepfakes and Synthetic Content

Deepfakes and AI-generated content have emerged as some of the fastest-growing trust and safety concerns for social media platforms. 

According to Ryabova, the challenge stems from the speed at which synthetic content can be created and distributed compared to the time required to verify and remove it. 

A convincing deepfake can be produced in seconds, while debunking it may take days or weeks.

The rise of open-source AI tools has lowered the barrier to entry for bad actors, enabling the creation of fake identities, manipulated images, voice clones, and fraudulent content at scale. 

These threats extend beyond misinformation and now include financial scams, sextortion, non-consensual intimate imagery, and political manipulation.

Identity Verification as a Defense Strategy

Rather than relying solely on deepfake detection, some social media platforms are adopting identity verification measures designed to confirm that users are real individuals. 

Ryabova said Wizz uses age verification and facial-matching technologies as part of a broader effort to reduce fake accounts and synthetic identities. 

She also noted that more than 40,000 accounts are permanently banned each year for failing verification requirements. 

Additionally, approximately 7% of age verification attempts are rejected due to authenticity concerns. 

By limiting access to verified individuals, the platform reduces opportunities for automated accounts and synthetic identities to spread harmful content.

AI Versus AI in Content Moderation

As generative AI becomes more sophisticated, social media platforms are increasingly relying on AI-powered moderation systems to identify harmful content. 

Wizz currently uses multiple layers of protection, including automated image and video moderation, behavioral monitoring, and human review.

The company is also developing an internal deepfake detection system specifically designed to identify manipulated profile images. 

Ryabova described this effort as a continuing “AI versus AI” battle, where defensive technologies must evolve alongside emerging threats.

While AI moderation improves scalability, it can still misread context and incorrectly flag content. 

Ryabova highlighted an example in which a user discussing hair dye was mistakenly flagged for self-harm concerns. 

Such incidents reinforce the importance of human oversight in moderation decisions.

The Impact of the TAKE IT DOWN Act

The Tools to Address Known Exploitation by Immobilizing Technological Deepfakes On Websites and Networks (TAKE IT DOWN) Act introduces new responsibilities for social media companies. 

One of its most notable requirements is the obligation to remove reported non-consensual intimate imagery, including AI-generated deepfakes, within 48 hours of receiving a valid complaint.

According to Ryabova, the law also requires platforms to implement systems that prevent previously removed content from being reuploaded. 

This proactive approach shifts responsibility from victims to platforms and establishes a higher standard for content moderation.

While the 48-hour timeline may be challenging for smaller organizations with limited resources, Ryabova believes companies that prioritize safety from the beginning are better positioned to comply. 

Investments in AI moderation, trusted safety partners, and industry collaboration can help organizations meet regulatory expectations.

Looking Ahead

As public awareness of AI-generated threats increases, users are demanding stronger privacy protections, identity verification, and platform accountability. 

Ryabova believes this trend is positive and reflects growing expectations for online safety.

The TAKE IT DOWN Act could serve as a model for future AI regulations by emphasizing rapid response, proactive detection, and greater platform accountability.