Fighting GenAI with GenAI: The New Email Security Landscape

The use of phishing emails as a means of stealing information or implanting damaging malware dates back to the mid-1990s.

Although almost as old as the Internet, it remains the instrument of choice for threat actors wanting to pose as trusted organizations or individuals for the purposes of corporate infiltration. In fact its use is on the rise: according to the most recent figures from the Federal Bureau of Investigation (FBI), some 26 percent of all cybercrime complaints filed with them are now phishing-related.

The bad news doesn’t stop there. Phishing is mutating in alarming new ways, raising all sorts of difficult questions for defenders and placing email security at a tricky inflection point. AI has for years been a useful tool in the hands of the cybercriminal. But the advent of Generative AI (GenAI) is proving a cyber game changer, transforming phishing from a widespread but easily identifiable nuisance into a lethal precision instrument.

Aspiring phishers have historically been hindered by various constraints. Their attempts at chummy authenticity have often been undermined by easy-to-spot mistakes and amateurish formatting – blemishes that traditional email security is good at picking up. Personalised attacks have also been hard to launch at any kind of scale. GenAI sweeps all technical, linguistic, and operational restrictions aside by automating sophistication.

Attackers no longer have to choose between individually targeted “spear phishing” which takes much effort to coordinate, and large-scale attacks that lack finesse. Now, with minimal expertise and at low cost, they can hit thousands of victims with minutely tailored phishes at the press of a button. Today’s Large Language Models (LLMs) generate polished and contextual text in any language, backing it with realistic brand graphics and convincing web addresses. At a stroke, Gen AI has democratized this type of crime, putting it in reach of anyone capable of the most elementary research.

“Thanks to AI, the historic signals that exposed a phishing email – poor grammar, misspelt words – are now ironed out and replaced with perfect language,” notes Dave Baggett, SVP Cybersecurity with software developer Kaseya. “These emails can be highly targeted with the help of AI. You can create an email that pinpoints, say, a pharma company by using authentic terminology. It’s an incredibly powerful tool for the bad guys.”

And let’s not forget that where this new AI-enabled phishing is successful, it is not just a single unwary employee whose data is at risk. Once past the perimeter, a bad actor can get to work burrowing into the cloud and SaaS platforms that now underpin global commerce and communications. All in all, it’s easy to understand how losses from phishing attacks have gone up 274 percent in the last couple of years, according to the FBI.

This new email security landscape is an alarming development for corporate IT bosses, and also for MSPs who must be super wary on behalf of customers who don’t have the expertise to pick up attacks at this pitch of sophistication on their own.

AI is for the good guys too

It’s not all bad news on the email security front line. As threat actors have been perfecting their use of GenAI, defenders have been developing AI-driven counter solutions in parallel. Where old school protection relied on spotting the kind of anomalies that GenAI has now ironed out, newer solutions are geared more towards contextual analysis. They are not so much trying to spot clumsy typos and dodgy attachments as model an attacker’s intent before damage is done. Today defenders can use AI to analyze subtle patterns that are in tune with the layered structure of modern phishing campaigns. The aim is to determine whether a message aligns with expected behavioral patterns.

This level of finesse is made necessary by the insidious nature of modern phishing which means that technically clean email messages, in other words ones that are free of malware-loaded attachments or booby-trapped links, can still contain malicious potential. Modern cybercriminals often bypass automated security filters by the simple means of exploiting human psychology and trusted infrastructure. Social engineering, supercharged by GenAI, can be a sharper sword than a malware payload.

A phish will often seek to impersonate a boss or a colleague using a plain text email. Attackers can set GenAI to scour LinkedIn, looking for people who have just started new jobs. Green employees are more vulnerable to a phish that looks like an email from a new superior they haven’t got to know yet.

Once trust is established, what will typically follow is a demand for, say, an urgent wire transfer, or a casual request for payroll details or passwords. Attackers can develop a relationship over multiple emails to “groom” the victim before striking. Traditional filters see nothing suspicious. Spotting this sort of attack demands a smarter grade of tool.

“Since attackers are relying on manipulation of fallible individuals, security must now support them at the moment where wrong decisions can be made,” says Baggett. “Good email security will replace generic warnings with easily digestible context about why a message might be risky and advice about the need for further verification.”

Where once an inbox was a passive delivery channel, says Baggett, it is now an active layer of risk mitigation, tooled up to stamp out a phishing attempt before it escalates into account compromise and devastating financial loss.

A huge asymmetry

The good guys are fighting back. But as Baggett points out, there remains a huge asymmetry between bad actor and defender: “Attackers can use LLMs, basically for free, to create perfect phishing templates,” he says. “On the other hand, if defenders took every inbound email and put it through a frontier LLM model, that’s about 100x too expensive. We just can’t use the same tools at the criminals.”

One option for defenders is to use smaller, more specialized models, distilled from larger ones. A compact model could be tuned, for example, to the sentiment or meaning of each phrase in an email, and set to pick up a threatening tone or a favor being asked. Where chunky LLMs just cost too much, another choice might be to run a subset of the overall inbound mail through a big model: “Admins can be given a way to run particular mails through a frontier LLM,” says Baggett. “That model will have been trained to understand certain critical aspects of email security.”

Better use could be made of pre-LLM tools such as computer vision, which can help process, analyze and understand visual data like digital images and videos. Other ancillary tools that could enrich the defensive mix include sender analysis to look into the origin and authenticity of an email message, and linked-content inspection that can examine an email in depth without triggering any embedded threats.

There are some reasons to hope for a safer future as AI technology evolves. As inference becomes smarter and models get better, it should become possible to run a higher percentage of emails through a large scale LLM, believes Baggett. This, however, may not be achievable for some years. There’s also the possibility of using LLMs not just to spot potentially malicious emails but also to analyze the settings and admin controls of existing security tools. In this way, the knowledge required to use the tools to best effect becomes much less.

 “Security tools can be complex and feature hundreds of settings,” points out Baggett. “Some require expertise that our customers don’t have. There are a number of things we believe tools should be doing to help defenders, for example reduce alert fatigue.”

The criminal fraternity may, of course, also find ways to harness better tools for new purposes. At the moment, most attacks are conceived and designed by humans. But ultimately, this job may be taken over by agentic models able to figure out new tactics for themselves. Machines may soon be able to plan and supervise attacks at scale.

Great tools, available now

Luckily defenders don’t have to sit back and be at the mercy of future developments. There are some great tools available today that help them even up the odds and cut the complexity from checking email. INKY Smart Insights, for example, uses GenAI to reduce a lengthy manual email investigation to a few seconds of automated triage. For the purposes of hard-pressed administrators it can turn a mess of technical email evidence into a plain-language verdict and enable the creation of cogent reports. Along with Kaseya Intelligence it can support decision-making across the broader security environment.

“Smart Insights will identify problems and give a narrative explanation of its reasoning,” enthuses Baggett. “It will do that in a way that’s not overly technical, making the tool valuable to someone who doesn’t have expertise in email security.”

The hard-pressed MSP, for example, can leverage Smart Insights to provide key information for their customers in an abbreviated and digestible format. In short, it allows them to enjoy the sophistication of LLMs but without adding hugely to their overheads.

The war on GenAI-powered phishing is far from won. But so long as defenders can understand the potential of the smart tools at their disposal, they have a fighting chance of keeping the bad guys at bay and essential channels of communication safe.

Find out more at Kaseya’s Cybersecurity Summit – details here

Sponsored by Kaseya