Building Resilient Distributed Systems: Failures, Circuit Breakers & Retries
In distributed systems, failures are guaranteed to happen. Networks partition, downstream third-party APIs experience degraded latency, and databases reach IOPS ceilings. Resiliency isn't about avoiding failures—it's about containing blast radiuses.---1. The Danger of Naive Retries (The Retry Storm)
When an upstream service times out and 10,000 clients simultaneously retry their requests every 500ms, they create an accidental self-inflicted DDoS attack known as a Retry Storm.
typescript
// ❌ Naive Retry (Destroys recovering downstream servers)
async function fetchWithNaiveRetry(url: string, attempts = 3) {
for (let i = 0; i < attempts; i++) {
try {
return await fetch(url);
} catch (err) {
await sleep(1000); // Fixed interval creates synchronized traffic spikes!
}
}
}2. Exponential Backoff with Full Jitter
Full jitter decorrelates retry intervals across all connected clients, smoothing out the traffic wave:
typescript
// ✅ Exponential Backoff with Full Jitter
async function fetchWithJitteredBackoff(
url: string,
maxRetries = 4,
baseDelayMs = 200,
maxDelayMs = 4000
) {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const res = await fetch(url, { signal: AbortSignal.timeout(3000) });
if (!res.ok && res.status >= 500) throw new Error(Server error: ${res.status});
return await res.json();
} catch (err) {
if (attempt === maxRetries - 1) throw err; // Exponential cap
const exponentialDelay = Math.min(maxDelayMs, baseDelayMs * Math.pow(2, attempt));
// Full Jitter formula: random between 0 and exponentialDelay
const jitteredSleep = Math.random() * exponentialDelay;
await new Promise(r => setTimeout(r, jitteredSleep));
}
}
}3. Circuit Breaker State Machine
A Circuit Breaker stops making outbound requests when failure rate exceeds a threshold:
mermaid
stateDiagram-v2
[*] --> Closed
Closed --> Open: Failure Rate > 50%
Open --> HalfOpen: Cooldown Timer (30s)
HalfOpen --> Closed: 5 Successes
HalfOpen --> Open: Single Failure