Social Engineering in the Age of Synthetic Media

Malicious Models, Phishing-as-a-Service (PhaaS), and Illegitimate Uses for Legitimate AI Tools

Social engineering refers to attempts to manipulate a person into sharing information, sending money, granting access, or acting against their own or their organization’s best interests. Threat actors have adopted legitimate and jailbroken large language models (LLMs) and generative AI (genAI) platforms to create, personalize, and scale social engineering campaigns more quickly and efficiently.

Examples include employing genAI to write and personalize phishing messages, research targets, translate content, analyze stolen inboxes, and build fake websites or login pages. Threat actors have also used genAI to automate follow-up messages and support employment, customer service, and business email compromise scams. When attackers use this deception to obtain money, access, services, or another benefit, it becomes fraud. Much of this activity involves phishing, a form of social engineering that uses false messages, websites, or sign-in requests to prompt an unsafe action. Phishing often begins the fraud by prompting the target to send money, disclose information, or surrender an account.

In addition to these malicious models, threat actors have been observed using legitimate AI tools to build phishing infrastructure. In August 2025, Proofpoint reported that threat actors used Lovable, a legitimate AI website builder, to create phishing pages and malware. Since its inception, Lovable has been used to create tens of thousands of malicious websites, including pages that impersonate Microsoft, UPS, and various human resources and financial services platforms.

Similarly, GLM-5.2, a model released by Chinese artificial intelligence company Z.ai (formerly Zhipu AI) in June 2026, was marketed as a tool to support autonomous AI agents in coding and design tasks. As an open-weight model, users can access GLM-5.2 privately and alter its programmed behavior outside the developer’s systems, reducing the developer’s visibility into harmful use.

As a result, security researchers warn that these same capabilities allow threat actors to carry out malicious AI-enabled activity, such as running private agentic workflows, developing offensive code, identifying vulnerabilities, and accelerating other malicious operations without detection and interference from the developer. Users on Russian-language forums have discussed altering the model and claimed to have used it to produce harmful content. While analysts have not confirmed GLM-5.2’s use in real-world attacks, other open-source, open-weight, models have been employed in the wild. According to Recorded Future research, The Gentlemen ransomware group used Chinese models, including Kimi, DeepSeek, Qwen, and HUIHUI-AI, to automate payload generation and technical analysis and to triage exfiltrated personally identifiable information (PII).

Threat actors have also expanded upon these standalone tools to offer packaged services, including phishing tools and infrastructure, to assist customers in carrying out social engineering campaigns. For example, EvilTokens, a phishing service that emerged in April 2026, reflects the professionalization of the phishing-as-a-service (PhaaS) ecosystem. Rather than offering only phishing templates or stolen login pages, PhaaS offerings such as EvilTokens combine AI-generated lures, research capabilities, account validation, and customizable phishing infrastructure within a single product. In one documented instance, EvilTokens automated the creation of fresh sign-in codes and used trusted cloud services and Microsoft’s official login page, which helped customers bypass security measures such as login time limits.

These AI-enabled tools and services should be taken seriously because of their ability to increase campaign volume, improve the personalization of phishing messages, and lower the skill and infrastructure barriers to staging social engineering campaigns; however, current evidence does not suggest that these AI enhancements and capabilities have rendered layered existing defenses ineffective.

Academic and industry studies find that AI-generated phishing messages perform within roughly the same range as human-written phishing messages, with results varying slightly by model, research context, and level of personalization (1, 2, 3, 4, 5). Layering existing controls remains broadly effective because they focus on suspicious senders, links, login requests, payment instructions, and user behavior rather than the author of the message. AI therefore increases exposure by producing more credible attempts at greater volume, but it does not consistently defeat the technical and procedural safeguards that already address phishing and other forms of social engineering.

To combat these AI-enabled social engineering threats, organizations should maintain strong email and web filtering, require phishing-resistant multifactor authentication (MFA), and monitor for unusual sign-ins, new inbox rules, unexpected account permissions, and suspicious use of newly created or hosted websites. Security teams should additionally quickly revoke active sessions and reset credentials after suspected compromise.

Organizations should also train employees not to treat polished writing, personalized details, or familiar branding as proof that a request is legitimate. Defenders should verify unusual requests through a separate trusted channel and report suspicious messages without interacting with them. Employee training should additionally include current examples of AI-generated and -polished lures, fake device sign-in requests, and AI-generated phishing pages.

Although layering existing defenses remains broadly effective against many forms of AI-enabled social engineering, synthetic media presents an exception.

Synthetic media includes audio, video, images, or documents that AI creates or alters. Attackers can use this content in two related ways. First, when they use a false voice, face, or document to manipulate a person, synthetic media becomes a tool for social engineering and often supports fraud. Second, when they send false biometric evidence directly to an identity system, they attack the identity check itself, even when the end user does not see the fraudulent content.

Threat actors have used AI to clone the voices of executives and officials, impersonate people during live video calls, alter their appearance during job interviews, falsify identification documents, and insert artificial faces or voices directly into identity-verification systems.

These techniques allow threat actors to manufacture the signals that defenders and security teams often treat as proof of identity, including a known voice, a visible face, or a live response. However, research indicates that human judgment and technical detection struggle to distinguish synthetic content from authentic interactions, making defenses that depend on detecting the manipulation unreliable on their own.

Threat actors have used AI-generated images, deepfake videos, and synthetic audio to impersonate trusted figures on live calls, recorded messages, or identity verification processes. In July 2026, LexisNexis Risk Solutions reported a 180% year-on-year increase in attacks involving deepfake documents, images, and videos designed to imitate a live person. For example, in February 2024, threat actors used deepfake video and synthetic voices to impersonate the chief financial officer and other employees of UK-based engineering firm Arup, persuading a legitimate employee to transfer approximately $25 million to the threat actor.

In December, the US Federal Bureau of Investigation (FBI) also warned that threat actors had used synthetic voice messages to impersonate multiple senior US officials between 2023 and 2025. Furthermore, in May 2026, threat actors employed the same approach to fabricate a Zoom meeting with Singapore’s prime minister, president, government officials, and private-sector representatives. The victim then transferred approximately SGD 4.9 million to the threat actors following the fraudulent meeting.

Synthetic media also poses a risk beyond traditional social engineering with biometric injection attacks, an attack type in which threat actors exploit a system rather than a person. This use case falls outside the strict definition of social engineering, but it warrants inclusion in this report because it uses the same synthetic content to undermine the signals that people and systems treat as proof of identity.

In both cases, the threat actor succeeds by making false evidence appear to confirm a real identity. In these attacks, threat actors stage biometric injection attacks to evade existing security and identity verification controls. In these attacks, threat actors insert altered or artificial faces directly into a video stream through virtual cameras or modified software to bypass the physical camera and live identity checks (Figure 3).

For example, between January and August 2025, Group-IB recorded 8,065 attempts to bypass an unnamed financial institution’s facial-liveness checks during digital loan applications. Threat actors used virtual cameras to insert AI-generated faces directly into the video stream presented to the institution’s identity system. Similarly, in April 2026, police in Ahmedabad, Gujarat, India, disrupted an identity-fraud operation that used AI to convert victims’ photographs into synthetic videos of them blinking. The threat actors presented these screen-replay videos to web-facing digital cameras to pass liveness checks, change victims’ registered phone numbers, access financial services, and obtain fraudulent loans.

Like the market for phishing services, an ecosystem around biometric injection attacks and live identity evasion has similarly emerged online. An April 2026 MIT investigation identified public Telegram groups selling deepfake tools designed to bypass identity checks, such as virtual-camera tools, modified mobile applications, and stolen biometric packages. According to the report, sellers advertised tools designed to target Binance, BBVA, Revolut, and other services.

Traditional email filters and general phishing training are inadequate to protect against this false biometric evidence or convincing real-time impersonation. Email filters examine messages, links, and attachments, but they cannot assess content delivered through video calls, voice calls, or directly to identity verification systems. Similarly, standard phishing training asks employees to look out for usual behavior or errors in speech or written text, but synthetic media reduces those warning signs by impersonating known individuals or by providing real-time responses. As a result, even after undergoing standard phishing training, employees are frequently unable to reliably distinguish authentic voices, images, and videos from convincing synthetic content. For example, a 2024 study titled “As Good As a Coin Toss” found that people identified synthetic images, audio, and video with only 51.2% accuracy.

Deepfake detection software is also inadequate as a standalone defense strategy, as recent studies indicate that deepfake detectors fail to reliably detect fraudulent content. A study of deepfakes collected from May 2026 found that even the strongest open-source deepfake detector performed only slightly better than statistical chance, demonstrating that organizations cannot rely on detection alone. Given these results, organizations cannot solely rely on individual employees or detector software to identify every instance of impersonated content.

Instead, organizations should use deepfake detection technology alongside procedures that require employees to verify sensitive requests through separate channels, such as calling a known number or messaging through an approved company system. Organizations should also require independent approval for payments, account changes, and access requests.

Organizations should additionally require a second approver for large payments, payroll changes, new access, account recovery, and changes to contact details, as well as pausing unusual or urgent requests until staff complete a separate check. Employees should be required to follow these same security protocols even when a senior leader makes the request, since threat actors often impersonate senior officials to leverage authority and urgency in order to override identity checks and approval requirements.

Outlook

Over the next six months to one year, synthetic media is likely to remain the area of AI-enabled social engineering that requires the greatest defensive adaptation. Threat actors are almost certain to continue using deepfake audio and video and biometric injection techniques, while uneven adoption of reliable deepfake detection, secure identity-capture systems, and independent verification processes will leave many organizations exposed. This assessment would shift if defenders adopted these controls broadly enough to substantially reduce the risk posed by manipulated audio, video, and identity evidence, but uneven detection performance and slow organizational adoption make that unlikely within the next year.

The broader outlook for AI-enabled social engineering would also shift if AI systems become consistently capable of conducting fully or mostly automated campaigns that can defeat filters, phishing-resistant MFA, training, and approval controls. Recent AI Security Institute testing demonstrates that frontier AI agents are developing autonomous social-engineering capabilities. In a July 2026 investigation, agents created false identities and attempted to pressure a GitHub maintainer into approving malicious code without being specifically instructed to do so. Although this attempt failed, it indicates that agents tasked with hacking into a network will use social engineering as a tactic. In the near term, this may result in high volumes of nuisance-level activity, which can be prevented by the same layered defenses effective for most human-generated efforts. However, the scale and frequency of these attempts may enable them to achieve occasional success, resulting in threat actors continuing to pursue automated social engineering capabilities.

About Insikt Group^®

Recorded Future’s Insikt Group, the company’s threat research division, comprises analysts and security researchers with deep government, law enforcement, military, and intelligence agency experience. Its mission is to produce intelligence that reduces risk for customers, enables tangible outcomes, and prevents business disruption.