Introduction
This is the fourth part of a series of blog posts about integrating event-driven microservices with request/response APIs. In the previous part, we discussed the use of circuit breakers to handle unavailability of request/response APIs. Now, we discuss how to adhere to usage limits of request/response APIs by using rate limiting.
Lesson learned #4: Rate limit the event processing
Request/response APIs that are provided by external service providers often have a usage limit, e.g. 100 requests per second, which upstream microservices must adhere to. Event-driven microservices that integrate with such request/response APIs may exceed their usage limits without additional measures.
Rate limiting is a useful way of ensuring usage limits aren’t exceeded. Typically, rate limiting is used by downstream microservices to protect them from too many requests. One way this can be achieved is by installing a web application firewall which blocks requests when an upstream microservice exceeds the usage limit. The web application firewall would then respond with an error such as 429 TOO_MANY_REQUESTS.
From the perspective of an event-driven microservice that sends requests to the request/response APIs, this type of error response would lead to retries. As with the examples in earlier blog posts, events would eventually end up in a dead letter queue.
Integrating a circuit breaker, as we discussed in part three of this series, can only help to a limited extent. As soon as the event-driven microservice exceeds the usage limit and receives the 429 TOO_MANY_REQUESTS errors, the circuit breaker would pause event processing; it would resume only after some time. If the waiting time before the circuit breaker resumes the processing is long enough, the web application firewall and the request/response API will accept requests again. However, the usage limit will likely be exceeded again since the circuit breaker doesn’t impact the speed of event processing. The circuit breaker will stop and resume the event processing again and again.
We have had a positive experience adding a rate limiter to the event-driven microservice so it does not exceed the usage limit of the API. Depending on the specific rate limiting implementation, different algorithms are offered. For example, the resilience4j library provides implementations of the token bucket algorithm with which scenarios like a maximum of 100 requests per second can be accomplished. This would allow you to configure a bucket with a maximum size of 100 tokens and the rate limiter to fill the bucket up to 100 tokens every second.
During event processing and before you send a request to an API with a usage limit, you use the rate limiter to acquire one token from the bucket. If there are enough tokens in the bucket, the number of available tokens is reduced by one and event processing can continue. If no tokens are available, the event processing can only continue later. The rate limiter can block the event processing until the bucket is refilled and tokens are available again.
The code sample below illustrates the integration of a rate limiter into event processing. The method blockAndAcquireToken() corresponds to the behavior described before. The method tries to acquire a token from the bucket; if a token is available, the token is removed from the bucket and event processing continues by creating and sending a request to the API. If no token is available, the method waits until new tokens are available. It blocks the event processing to avoid exceeding the API’s usage limit.
void processEvent(Event event) {
/* ... */
rateLimiter.blockAndAcquireToken();
Request request = createRequest(event);
Response response = sendRequestAndWaitForResponse(request);
moreBusinessLogic(event, response);
/* ... */
}
Listing 1: Event Processing with a Rate Limiter
With implementations like resilience4j, the blockAndAcquireToken() method only blocks processing for a specified amount of time before it returns an error and fails the event processing. The configured wait time and the middleware’s visibility timeout should be aligned. If the blockAndAcquireToken() method blocks event processing because no tokens are available, it should return an error and fail the event processing before the visibility timeout elapses. Further on, if it acquires a token and resumes the event processing, there should be enough time to finish the processing of the event before the visibility timeout elapses.
If this is not possible, it results in duplicate processing of events. Although we discussed how to deal with duplicate events in the first blog post, they should be avoided by configuring a visibility timeout and the wait time of the rate limiter accordingly.
Conclusion
External service providers often define and enforce usage limits on the request/response APIs they provide. In this blog post, we’ve seen how rate limiting can be a useful solution to adhering to usage limits in event-driven microservices.
This blog post concludes this series about our experience integrating event-driven microservices with request/response APIs. As we’ve seen, integration involves some considerable challenges. However, in many cases integration can’t be avoided — if you rely on third party providers or on systems that haven’t been migrated to an event-driven architecture as part of a legacy modernization, it’s something you are going to need to do.
To finish, our experience taught us four key things that we think you can benefit from:
Retries are inevitable and event processing including the request/response APIs should be idempotent (Part One).
The response times of request/response APIs impact the performance of event-driven microservices. If response times fluctuate significantly, it makes sense to decouple event retrieval from event processing (Part Two).
Circuit breakers can improve the integration with request/response APIs as they handle periods of unavailability (Part Three).
Request/response APIs may have usage limits and rate limits help adhering to them (Part Four)
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
