SYDNEY, AUS. – Nov. 29, 2023 – Sophos, a global leader in innovating and delivering cybersecurity as a service, today released two reports about the use of AI in cybercrime. The first report—“The Dark Side of AI: Large-Scale Scam Campaigns Made Possible by Generative AI”—demonstrates how, in the future, scammers could leverage technology like ChatGPT to conduct fraud on a massive scale with minimal technical skills. However, a second report, titled “Cybercriminals Can’t Agree on GPTs,” found that, despite AI’s potential, rather than embracing large language models (LLMs) like ChatGPT, some cybercriminals are sceptical and even concerned about using AI for their attacks.

The Dark Side of AI

Using a simple e-commerce template and LLM tools like GPT-4, Sophos X-Ops was able to build a fully functioning website with AI-generated images, audio, and product descriptions, as well as a fake Facebook login and fake checkout page to steal users’ login credentials and credit card details. The website required minimal technical knowledge to create and operate, and, using the same tool, Sophos X-Ops was able to create hundreds of similar websites in minutes with one button.

“It’s natural—and expected—for criminals to turn to new technology for automation. The original creation of spam emails was a critical step in scamming technology because it changed the scale of the playing field. New AIs are poised to do the same; if an AI technology exists that can create complete, automated threats, people will eventually use it. We have already seen the integration of generative AI elements in classic scams, such as AI-generated text or photographs to lure victims.

“However, part of the reason we conducted this research was to get ahead of the criminals. By creating a system for large-scale fraudulent website generation that is more advanced than the tools criminals are currently using, we have a unique opportunity to analyse and prepare for the threat before it proliferates,” said Ben Gelman, senior data scientist, Sophos.

Cybercriminals Can’t Agree on GPTs

For its research into attacker attitudes towards AI, Sophos X-Ops examined four prominent dark web forums for LLM-related discussions. While cybercriminals’ AI use appears to be in its early stages, threat actors on the dark web are discussing its potential when it comes to social engineering. Sophos X-Ops has already witnessed the use of AI in romance-based, crypto scams.

In addition, Sophos X-Ops found that the majority of posts were related to compromised ChatGPT accounts for sale and “jailbreaks”—ways to circumvent the protections built into LLMs, so cybercriminals can abuse them for malicious purposes. Sophos X-Ops also found ten ChatGPT-derivatives that the creators claimed could be used to launch cyber-attacks and develop malware. However, threat actors had mixed reactions to these derivatives and other malicious applications of LLMs, with many criminals expressing concern that the creators of the ChatGPT imitators were trying to scam them.

“While there’s been significant concern about the abuse of AI and LLMs by cybercriminals since the release of ChatGPT, our research has found that, so far, threat actors are more sceptical than enthused. Across two of the four forums on the dark web we examined, we only found 100 posts on AI. Compare that to cryptocurrency where we found 1,000 posts for the same period.

“We did see some cybercriminals attempting to create malware or attack tools using LLMs, but the results were rudimentary and often met with scepticism from other users. In one case, a threat actor, eager to showcase the potential of ChatGPT inadvertently revealed significant information about his real identity. We even found numerous ‘thought pieces’ about the potential negative effects of AI on society and the ethical implications of its use. In other words, at least for now, it seems that cybercriminals are having the same debates about LLMs as the rest of us,” said Christopher Budd, director, X-Ops research, Sophos.

For more about AI-generated scam websites and threat actors’ attitudes to LLMs, read The Dark Side of AI: Large-Scale Scam Campaigns Made Possible by Generative AI and Cybercriminals Can’t Agree on GPTs on Sophos.com.

— END

Learn More About:

• The ways in which defenders can use AI as a cybersecurity co-pilot
• The latest tactics deployed by pig butcherers, including the use of generative AI
• Scammers scamming other scammers on the dark web
• Cybercrime contests on the dark web
• How defenders can combat attackers in a fast-moving threat landscape in the 2023 Active Adversary Report for Security Practitioners
• Dwindling dwell times and changing attacker behavior and techniques in the Active Adversary Report for Tech Leaders 2023
• Changing attacker behaviors, techniques and tactics in the 2023 Active Adversary Report for Business Leaders, based on an analysis of more than 150 Sophos incident response cases
• Sophos X-Ops and its groundbreaking threat research by subscribing to the Sophos X-Ops blogs

About Sophos

Sophos is a worldwide leader and innovator of advanced cybersecurity solutions, including Managed Detection and Response (MDR) and incident response services and a broad portfolio of endpoint, network, email, and cloud security technologies that help organisations defeat cyberattacks. As one of the largest pure-play cybersecurity providers, Sophos defends more than 500,000 organisations and more than 100 million users globally from active adversaries, ransomware, phishing, malware, and more. Sophos’ services and products connect through its cloud-based Sophos Central management console and are powered by Sophos X-Ops, the company’s cross-domain threat intelligence unit. Sophos X-Ops intelligence optimises the entire Sophos Adaptive Cybersecurity Ecosystem, which includes a centralised data lake that leverages a rich set of open APIs available to customers, partners, developers, and other cybersecurity and information technology vendors. Sophos provides cybersecurity-as-a-service to organisations needing fully managed, turnkey security solutions. Customers can also manage their cybersecurity directly with Sophos’ security operations platform or use a hybrid approach by supplementing their in-house teams with Sophos’ services, including threat hunting and remediation. Sophos sells through reseller partners and managed service providers (MSPs) worldwide. Sophos is headquartered in Oxford, U.K. More information is available at www.sophos.com