Spring Cloud Gateway · Production Architecture
When a correct rate limiter becomes an incorrect architecture — and how shared state, Redis, and layered edge protection solve the hardest distributed systems problems.
We design and implement production-grade API gateway architectures with distributed rate limiting, observability, and resilience built in from the start.
Talk to Our TeamRate limiting is only one component of a layered traffic protection strategy. Our request path is designed so each layer has a distinct, non-overlapping responsibility.
Most engineering teams encounter rate limiting early in their API journey. The implementation often starts with a familiar pattern:
private final ConcurrentHashMap<String, TokenBucket> buckets =
new ConcurrentHashMap<>();For a single application instance, this solution is simple, fast, and effective. The problem appears when the platform evolves — load balancers, autoscaling, Kubernetes, multiple availability zones, and eventually multiple gateway replicas. At that point, the challenge is no longer implementing a token bucket algorithm. The challenge becomes maintaining a consistent view of rate limits across a distributed system.
CloudFront provides global edge caching, reduced latency, reduced origin load, and additional DDoS protection. Traffic is filtered and optimized before it reaches our infrastructure.
AWS WAF protects against SQL injection attempts, cross-site scripting attacks, malicious bots, IP reputation threats, and excessive request rates. The goal is to stop unwanted traffic as close to the edge as possible.
Spring Cloud Gateway is responsible for authentication, authorization, request routing, request transformation, rate limiting, and observability. This is where application-level throttling occurs.
A rate limiter that is functionally correct can still be architecturally wrong once you add a load balancer.
Consider a limit of 100 requests per minute. A single gateway instance enforces this correctly.
Client
|
GatewayHowever, production environments rarely stay this simple. A more realistic deployment looks like:
+--> Gateway Pod 1
Load Balancer ----+--> Gateway Pod 2
+--> Gateway Pod 3Each gateway pod maintains its own memory. This means:
Gateway Pod 1 = 100 requests
Gateway Pod 2 = 100 requests
Gateway Pod 3 = 100 requestsA client can effectively consume 300 requests per minute without violating any local limit.
"The token bucket implementation is functioning correctly. The architecture is not. Every gateway has a different view of reality."
A great library, but designed for a different problem domain entirely.
One misconception we frequently encounter is: "Why not use Resilience4j RateLimiter?"
Resilience4j is an excellent library and remains an important part of modern Java architectures. However, its primary purpose is different. Typical use cases include protecting downstream APIs, limiting outbound requests, preventing resource exhaustion, and implementing retries and circuit breakers.
RateLimiter limiter =
RateLimiter.of("payment-provider", config);This works extremely well when protecting external dependencies. However, each application instance maintains its own limiter state:
Gateway Pod 1 -> Limiter A
Gateway Pod 2 -> Limiter B
Gateway Pod 3 -> Limiter CThere is no distributed coordination. This is not a limitation of Resilience4j — it is simply outside the problem domain it was designed to solve.
Resilience4j RateLimiter — protects outbound calls to external services. Per-instance, no coordination needed.
Distributed RateLimiter — enforces global inbound limits across all gateway replicas. Requires shared state.
After evaluating multiple options, Redis provided the best balance between performance, simplicity, and operational maturity.
| Requirement | Redis Capability |
|---|---|
| Shared state | All gateway replicas consult the same counters |
| Atomic operations | Rate-limit decisions remain consistent under high concurrency |
| Low latency | Sub-millisecond evaluation — no noticeable request overhead |
| Ecosystem maturity | Well-established pattern used across API gateways and service meshes |
| Spring integration | Native RedisRateLimiter in Spring Cloud Gateway |
Avoiding a custom distributed rate-limiting implementation was an important architectural decision.
Spring Cloud Gateway already provides:
RequestRateLimiter
RedisRateLimiterThis allowed us to use a proven implementation instead of building and maintaining our own distributed coordination mechanism. The RedisRateLimiter uses a token bucket algorithm implemented as an atomic Lua script — handling concurrency correctly without custom code.
"Whenever possible, platform capabilities should be preferred over custom infrastructure code. The maintenance burden of a homegrown distributed rate limiter is significant."
RedisRateLimiter is battle-tested across thousands of production deployments. Building an equivalent from scratch requires solving distributed counter atomicity, clock skew, TTL management, and connection pooling — problems already solved in the platform.
Different consumers require different controls. Production rate limiting should be layered by consumer type and endpoint sensitivity.
Limited primarily by IP address. Useful for blocking bots, crawlers, and brute-force attempts before they consume authenticated resources.
Limited by User ID, Tenant ID, or API Key. Supports fair usage policies, multi-tenant isolation, and per-customer abuse prevention.
Additional restrictions applied to high-risk paths:
/login
/register
/password-reset
/tokenThese endpoints have fundamentally different security requirements than standard business APIs. A credential-stuffing attack may only send 10 requests/minute per IP — well within typical API limits — but still cause enormous damage if those limits aren't tightened at the endpoint level.
/login require aggressive limits (e.g., 5 req/min) while bulk data APIs may legitimately need thousands.
One of the most important architectural discussions: what happens if Redis becomes unavailable?
Many distributed rate-limiting designs effectively turn Redis into a critical dependency. If Redis is down, requests are either denied entirely (fail-closed) or allowed entirely (fail-open). For customer-facing platforms, either extreme can create an unnecessary outage or security gap.
We deliberately chose a different approach — a degraded operating mode with local fallback.
A conscious tradeoff between global consistency and platform availability during Redis outages.
Gateway Pods
|
v
RedisRateLimiter (global shared state)Gateway Pods
|
v
Local Token Buckets (per-pod fallback)In degraded mode, requests continue flowing and local throttling remains active. What we lose is global consistency — each gateway replica temporarily enforces limits independently.
"For our platform, preserving customer availability was more important than maintaining perfectly synchronized rate limits during a Redis outage. That is a conscious tradeoff, not an oversight."
Fallback strategies are only valuable if teams know they are active.
We expose metrics through Prometheus and visualize them in Grafana. Key indicators include:
rate_limiter_allowed_totalrate_limiter_denied_totalrate_limiter_redis_errors_totalrate_limiter_fallback_activations_totalrate_limiter_degraded_mode_activerate_limiter_latency_msThis gives operators immediate visibility into Redis connectivity issues, increased traffic volume, abuse patterns, and gateway behavior changes.
rate_limiter_degraded_mode_active should trigger a PagerDuty alert. Degraded mode is not a silent fallback — it means Redis is unavailable and the team needs to act.
"Observability is not an afterthought. It is a core architectural requirement. A fallback that activates silently is a liability, not an asset."
Even during a Redis outage, multiple protection layers remain active.
This layered approach ensures that no single component becomes the sole line of defense. Rate limiting works best when combined with edge protection, security controls, and application-level safeguards.
Distributed rate limiting is not primarily a rate-limiter problem — it is a distributed systems problem.
The challenge is not implementing token buckets, counters, or throttling rules. The challenge is ensuring that multiple gateway instances make consistent decisions while preserving availability when dependencies fail.
| Layer | Role in Our Architecture |
|---|---|
| Spring Cloud Gateway | Enforcement layer — routing, auth, rate limiting |
| Redis | Shared distributed state for consistent limits |
| CloudFront + AWS WAF | Edge protection — DDoS, bot filtering, IP reputation |
| Local token buckets | Degraded-mode resilience when Redis is unavailable |
| Prometheus + Grafana | Operational visibility across all layers |
"In distributed systems, the most important architectural decisions are often not about algorithms. They are about tradeoffs — balancing consistency with availability while keeping operational complexity manageable."
RedisRateLimiter implements it correctly.Topics