How to tackle bots in REST APIs

Question

I have a mobile application where users give advertise, other users view and accept it. Recently, I began to notice that bots started to give their own advertisements. I have moderators but there a so much advertisements that it is impossible to check everything (another challenge, is that bargain happens instantly, in realtime). It is classical REST API. I googled a lot and to my surprise can't find any open source solution that protects from illegal bot activities. How do you tackle such cases? Is it possible to eliminate it at all or I can only make their life more difficult taking some measures?

Answer 1

OPEN SOURCE BOT DETECTION

I googled a lot and to my surprise can't find any open source solution that protects from illegal bot activities.

If you search Github for bot detection you will land on this page https://github.com/topics/bot-detection?q=bot+detection&unscoped_q=bot+detection, that at this moment contains 7 results, that aren't very relevant, but if you remove the query string, and use https://github.com/topics/bot-detection you get 36 results, where some may be relevant depending on your backend language. You can also search using the terms browser detection, crawler detection, device detection, etc.

Some of this repos rely on the user-agent and/or in the IP address to detect the bot, and this approach is easily bypassed, because the user-agent header is easy to spoof and nowadays the attackers are using bot farms to rotate the ips, thus making it very hard to block them.

But should you use one of this repos? Why not, it's one more layer of defense, and at least you block the less sophisticated bots that don't run from a bot farm.

The Difference Between WHO and WHAT is Accessing the API Server

Recently, I began to notice that bots started to give their own advertisements.

Before we dive in how you can tackle the problem I want to first clear a misconception that is usual among developers of any seniority level, the difference between Who vs What is accessing the API server.

I recommend you to read the article Why Does Your Mobile App Need An Api Key? where I go in detail about the difference between Who and What is accessing your API server, but for your convenience I will extract here the main takes from it:

The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?

The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.

So I want you to think about the Who as the user your API server will be able to Authenticate and Authorize access to the data, and think about the What as the software making that request in behalf of the user, that in your case are the bots.

THE API SERVER DEFENSES

How do you tackle such cases? Is it possible to eliminate it at all or I can only make their life more difficult taking some measures?

You can make the life of an attacker harder by applying defense in depth, by adding as many layers of defense as you can afford, and is required by law for your use case.

WAF - Web Application Firewall:

A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to and from a web application. A WAF is differentiated from a regular firewall in that a WAF is able to filter the content of specific web applications while regular firewalls serve as a safety gate between servers. By inspecting HTTP traffic, it can prevent attacks stemming from web application security flaws, such as SQL injection, cross-site scripting (XSS), file inclusion, and security misconfigurations.

The effectiveness, against APIs is weak, because it was designed more specifically for web apps that don't rely on API's, but still able to offer some degree of protection.

UBA - User Behavior Analytics:

User behavior analytics (UBA) as defined by Gartner is a cybersecurity process about detection of insider threats, targeted attacks, and financial fraud. UBA solutions look at patterns of human behavior, and then apply algorithms and statistical analysis to detect meaningful anomalies from those patterns—anomalies that indicate potential threats. Instead of tracking devices or security events, UBA tracks a system's users. Big data platforms like Apache Hadoop are increasing UBA functionality by allowing them to analyze petabytes worth of data to detect insider threats and advanced persistent threats.

A good example of a UBA solution is Recaptcha by Google, specially the reCAPTCHA V3:

reCAPTCHA is a free service that protects your website from spam and abuse. reCAPTCHA uses an advanced risk analysis engine and adaptive challenges to keep automated software from engaging in abusive activities on your site. It does this while letting your valid users pass through with ease.

...helps you detect abusive traffic on your website without any user friction. It returns a score based on the interactions with your website and provides you more flexibility to take appropriate actions.

When you use recaptcha V3 in a mobile app your API server can then verify the score for that request, but bear in mind that it only makes it more difficult to bypass, because if you search Google for bypass recaptcha V3 you will see a lot of solutions being offered as a service for attackers.

Lock the API server to the mobile app

It's possible for the API server to have a high degree of confidence that the request is indeed from your mobile app, and not from a bot by using the Mobile App Attestation concept, and I invite you to read my reply to the question How to secure an API REST for mobile app?, specifically the section A Possible Better Solution.

DO YOU WANT TO GO THE EXTRA MILE?

In any response to a security question I always like to reference the amazing work from the OWASP foundation.

For Mobile Apps

OWASP Mobile Security Project - Top 10 risks

The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.

OWASP - Mobile Security Testing Guide:

The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.

For APIS

OWASP API Security Top 10

The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.

Answer 2

Stopping bots is something we are already dealing with a lot. One of the common mechanisms is using captcha mechanisms (e.g. Googles recaptcha). Please google for it.