The bot landscape is changing. Malicious — or bad bots — are evolving to become more advanced and human-like in their behavior, while an emerging category of AI bots, which we might think of as “grey bots,” is blurring the boundary of legitimate activity.

Barracuda security researchers analyzed bot-related traffic and activity targeting web applications and APIs between September 2023 and the end of August 2024. Among other things, the researchers found that:

• Bad bots make up 24% of internet traffic in 2024, down from 39% in 2021.
• The number of individual bad bots has risen and now comprises 44% of detected clients, compared to 36% a year ago.
• 49% of bots are classed as ‘advanced bots,’ most of which are malicious and designed to mimic human behavior and handle complex online interactions such as engaging with targets in account takeover attacks.

The Bot Landscape in 2024

Bots are automated software programs designed to perform online activities at scale. Good bots include search engine crawler bots, SEO bots, and customer service bots that can help organizations streamline processes, increase efficiency, boost their online presence, and strengthen customer interactions.

Bad bots, on the other hand, are designed for malicious or harmful online activities. Bad bots can be deployed against many different targets, including websites, servers, application programming interfaces (APIs), and other endpoints. Bad bots target e-commerce and login sites, among others, with the aim of breaching accounts to steal personal data or commit fraud, and they exploit vulnerabilities in websites for access. Bad bots can overload the target with traffic, spread spam, skew business analytics, disrupt services for legitimate customers, harm business reputations, and more.

The Emergence of AI “Grey Bots”

Barracuda security researchers also noted an emerging category of AI bots, which could be classified as “grey bots” because they are not overtly malicious, but their approach can be questionable.

These AI bots are primarily designed to extract or scrape large volumes of data from websites, for example, to train generative AI models. The bots can be aggressive when collecting data and may remove information without permission, possibly ignoring any embedded robots.txt code that is added by publishers to signal to scraper bots that they shouldn’t take that website’s data.

The Changing Dynamic Between Bots and Humans

Traffic distribution – Bots vs. humans

From September 2023 to the end of August 2024, good bots accounted for 18% of internet traffic, while bad bots made up 24%, and human users 58%.

The proportion of bad bot traffic is declining year on year. In 2023, bad bots accounted for 30% of internet traffic, down from 39% in 2021.

On the surface, this seems like good news. However, a deeper analysis shows that while the proportion of bad bots has declined, the proportion of individual bad bots has risen over the last 12 months. In other words, there is less traffic on the road, but many more makes of vehicle.

The researchers believe that the general decline in bad bot traffic detections is driven both by growing awareness of the threat and reduced demand for mass automated shopping bots.

More companies are aware of the damage that bad bots can do to their web applications and are taking steps to detect and block malicious or suspicious bot traffic. This has reduced the success rate of automated bad bot attacks and made them less attractive to cyberattackers.

In 2021, bad bot traffic included swarms of shopping bots targeting e-commerce sites to grab high-value consumer items to resell at a significantly inflated price. This included the infamous “sneaker-bots” hunting limited edition shoes. When the market for such products collapsed during the economic downturn, the demand for mass shopping bots declined, reducing the volume of bad bot traffic.

In its place we now have more advanced and targeted bots.

Bad Bot Activity in 2024

Our security researchers also looked at the types of bot activity detected in the eight months between January and the end of August 2024.

The data shows that bot activity in 2024 is dominated by ‘advanced bots’ and that most of these are confirmed as malicious:

• Advanced bots: These account for 49% of bot activity, much of it malicious. The malicious bots use sophisticated techniques to mimic human behaviors, and they can navigate complex web interactions, bypassing standard controls that look at rate of traffic, error rate, CAPTCHA, and IP addresses. Examples include account takeover bots that use multiple methods to perform so-called ‘low and slow’ attacks, which leverage different IP/geo locations to stay under the radar and evade detection. It should be noted that some advanced bot classifications can be unintentional false positives, where benign web crawlers or other tools that have upgraded their capability to avoid being blocked are inadvertently mislabelled as malicious.
• Impersonator: As for advanced bots, these are bots designed to impersonate human behaviors, typically for malicious purposes such as fraud. An example of this would be a bot attempting to spoof the GoogleBot to avoid being blocked from scraping.
• Known violator: These are previously identified entities that have engaged in undesirable or malicious activity. For security tools, they represent bots that have been caught before and are therefore prevented from being able to access protected applications. Know violators accounted for 6% of activity in the period analyzed.
• Browser integrity anomalies: These are detected clients that have anomalies in their browser configurations that could suggest emulation or spoofing. They made up 3% of activity overall.

Good Bot Activity Detected

• Crawler/indexer: These bots are primarily used for indexing web content for search engines.
• Feed fetcher: Bots that retrieve content for web feeds, aggregators, or news crawlers.
• Search engine bot: Bots that interact with company systems for the purpose of indexing for search engine optimization.
• Social media agent: Automated agents that manage or interact through social media platforms.
• Technical partner/commercial bot: Bots operated by third-party companies for integrating services or content.
• Tool: Clients using tools for testing, monitoring, or other operational functions.

How to Protect Your Organization

Understanding and addressing the threat of bad bots is crucial for maintaining the security and integrity of online activities. This includes protecting e-commerce sites against price scraping, inventory hoarding, and fraudulent transactions, and preventing spam, fake accounts, and misinformation campaigns from targeting social media, as well as safeguarding proprietary data, protecting against negative SEO tactics that can harm website rankings, and ensuring that login authentication attempts are legitimate.

Effective, targeted bot protection helps to detect and protect against automated attacks carried out by malicious bots, while at the same time enabling known good bots, such as search engine crawler bots and SEO bots, to crawl your web application.

Such protection requires a multilayered approach, including:

• Robust application security. Install advanced application protection to protect web applications and APIs and make sure it is properly configured with rate limiting and monitoring in place. This is an important first step to make sure your application security solution is working as intended.
• Specialized bot protection. Make sure the application security solution you choose includes anti-bot protection to effectively detect and stop advanced automated attacks.
• Take advantage of machine learning. With a solution that uses the power of machine learning, you can effectively detect and block hidden, almost-human bot attacks. Be sure to turn on credential stuffing protection to prevent account takeover as well.
• Don’t forget the basics. Access and authentication controls, including multifactor authentication, will help to secure vulnerable access points such as login pages from brute force and credential stuffing attacks.

For more information on how to defend your environment against advanced and evolving bot attacks, visit our website.