How to rate limit outbound traffic to an external/3rd party resource?

Question

I'm running a distributed system where multiple services need to make API requests to an external, 3rd party resource. However, this 3rd party API has API limits (n requests per hour, m concurrent requests) which we need to adhere to (Note: if you reach n requests in an hour, then the API responds with an error, and the number of seconds to wait before retrying).

If there wasn't the concurrent requests requirement, we could obviously just look at the header and wait until the API limit was over. But, how can I ensure that 2 microservices don't exceed the concurrent request limit? My ideas were:

1. Create a centralized service that sends requests to the 3rd party API. The microservices submit requests to the centralized service, which handles API limits. The downside is that the API requests include large packets of data and large responses that would need to be routed through the centralized service

2. Create a light-weight, centralized lock of sorts that microservices call to get permission to use the API. The centralized service will ensure that only m concurrent requests are being sent at a time, and never hand out the lock if n requests have been reached that hour. I'm leaning toward this solution but might it be fiddly?

3. I wonder if there's some out-of-the-box infrastructure like a gateway for outbound calls that all requests to this 3rd party API are routed through. This is a random idea, I'm unsure if such infra exists

Curious to hear people's thoughts

Answer 1

What is the added value of setting up something on top of the 3rd party API? If services make too many requests, they will get an error anyway, whether directly from the 3rd party API or from your system, and they will have to retry their requests all the same.

If what you want is absorbing peaks of requests without reaching the limit of the 3rd party API, then you might want to set up a message bus to enqueue the requests and an agent that processes the messages at the rate of the 3rd party API. Be careful then that you properly handle edge cases : in some situations, you might have to ignore or drop some messages because the 3rd party API cannot cope with all the received requests that are queued into the message bus.