The bad guys use AI to commoditise Scams & Phishing. Defenders now use AI to fight back!

"Earning money online has never been so easy"

Most users expect security products to block all “unwanted or bad things”. However, security products have to deal with classifying unwanted messaging and treat them differently.

Security products have a new ally in dealing with Scams & Phishing, AI Assistants.

Not every disappointing online purchase constitutes a scam, while classic fraud - such as fake penalty payment demands - always qualifies. Users may not notice these nuances, but for security products accurate classification is essential for both effectiveness and user trust. Security products have a new ally in dealing with this conundrum, AI.

Protecting against phishing and scams is ever a more complex problem. As users we interact with the internet across almost all our devices. This makes the attack surface ever increasing. We exchange information with our devices across multiple apps, social media, banking and, job boards and so on. Attackers use this to their advantage. A threat chain is all too familiar, however a scam communication chain can jump between multiple applications and even devices. Security products therefore must keep an eye on multiple attack vectors, sometimes piecing them together for context.

These expectations often conflict with the technical realities of security products. Most individuals simply want all unwanted or potentially harmful messaged to be blocked, but the distinction between “scam” and “spam” isn’t always obvious. Technically identifying a scam is challenging, but the users don’t care for that.

We’ve evaluated a series of products taking advantage of AI to tackle Scams & Phishing. While we have our own rating attributed to the test scenarios presented we also shared all the raw results on our Github for readers who want to come up with their own rating. This is the first test of its kind in the industry. We understand that the methodology and scope isn’t perfect, we are actively seeking feedback from both vendors and practitioners in this area.

The 3 tested vendors were GenDigital, McAfee and Bitdefender. We would like to thank them for their participation and willingness to process the results from our lab. Introducing a new methodology to such a fresh sector is always tricky and we will never get everything right. The AMTSO standard was followed throughout the whole test. You can find the AMTSO tracking page here. For those unfamiliar with it, the standard pushes the testing industry for transparency and fairness in how testers operate. All tested participants were given notice of the test execution, the right of dispute and a preview of results in advance of publishing.

One of the major pieces of feedback we had after the first preliminary set of results is extension of some test scenarios to be submitted through their AI assistants. This extended our test deadline but since this was request that came from all 3 parties individually it was a fair way to extend the testing from the original exposure. You can find all raw results for the participants in our Github. You can also take the results and make your own rating system for a specific component that you may care about more.

Furthermore, we are working on improvements in terms of scope and test design for the next run of the test currently planned for Q1 2026. The key improvements we are looking to make for 2026 are:

·   Increased sample size

·   Increased component specific evasion techniques

·   Increased domain variety

·   Education rating consideration

·   More granular rating

There is still time to submit your thoughts on what you would like to see in this area. We are also active participants in the AMTSO Scam & Phishing working group. This is a joint effort by several security vendors and testing labs to come forward with a set of guidelines for tests operating in this area.

Working groups such as these is one of the biggest benefits of having a community such as AMTSO. It allows industry experts on a specific topic to push our industry forward with good tests and better security products for the end user. If you are interested in shaping the future of testing in this area please consider joining AMTSO and the working groups you are interested in.

For our methodology keep an eye out on for an announcement on the material changes in November.

You can view all the results and our transparency report over in our dedicated area for this testing.