Our Load Test Passed Because It Never Triggered Retries

How 300 customer requests became nearly 900 downstream calls—and how to load-test a Spring Boot API against a slow dependency.

Three hundred customer requests produced nearly 900 downstream calls.

The customers did not create most of that extra traffic. Our Spring Boot service did, through retries.

Our search API normally waited about 200 milliseconds for its downstream service.

Then the downstream latency jumped to four seconds. Search requests began timing out, the HTTP connection pool filled, and Tomcat threads remained busy far longer than normal.

The downstream slowdown started the incident. Our retry policy amplified it into a local capacity problem. And the load test we had confidently passed had never exercised that path.

A load test is incomplete if it measures normal traffic but never triggers the retries, timeouts, and resource exhaustion that appear when a dependency slows down.

The obvious suspect was the downstream service

The search API ran on Spring Boot 3.5.7 and called its dependency through OkHttp 4.9.2.

Our first theory was reasonable: the downstream service was unhealthy, and the search API was an innocent victim.

That theory was only half right.

The dependency really had slowed down. But incoming traffic had not increased enough to explain why the local connection pool was filling or why so many Tomcat request threads were occupied at the same time.

At 200 milliseconds per call, one connection can complete roughly five sequential calls per second. At four seconds per call, the same connection completes only one call every four seconds. That is a 20x increase in occupancy before considering a single retry.

The useful capacity question was not only how many requests arrived. It was how long each request occupied a thread and connection. A steady arrival rate can overload a service when every request holds those resources much longer.

The metric that changed the investigation

We compared two counts that had previously been treated as if they were the same:

• Customer requests entering the search API
• HTTP attempts leaving the search API

They were not the same.

Tracing and attempt-level logs showed that roughly 300 incoming requests had expanded into nearly 900 downstream attempts. In simple terms:

downstream call rate =
incoming request rate × average attempts per request

Three hundred requests at an average of three attempts produce about 900 calls.

This equation explains traffic amplification, but it understates the capacity problem. Slow attempts overlap. The useful concurrency approximation is:

in-flight downstream calls ≈ downstream call rate × attempt duration

Both terms had increased. We sent more attempts, and each attempt remained active longer.

That was the missing part of our load model.

How one slow call became a capacity problem

The incident sequence was straightforward once the attempt count was visible:

1. Downstream latency increased from about 200 milliseconds to four seconds.
2. Connections and Tomcat threads remained occupied roughly 20 times longer.
3. Calls timed out or otherwise qualified for a retry.
4. Retried calls reached the same degraded dependency.
5. New customer traffic continued to arrive.
6. The HTTP connection pool and downstream concurrency capacity filled.
7. Customers waited longer while the application created more failed work.

The incident record did not establish whether every retry ran on the original request thread. That is one of the first details to verify. In a synchronous design, backoff, connection-pool waiting, network execution, and retries all spend time from the same customer request thread.

That detail matters. Moving retries to another executor might free Tomcat threads, but it does not remove the work. It can merely move the queue and make the overload harder to see.

OkHttp also has its own retryOnConnectionFailure behavior. An application-level retry can therefore sit on top of client-level recovery unless both layers are audited. The actual attempt count must come from telemetry at the network-call boundary, not from the number of times an application retry callback fires.

Why missing jitter made the incident worse

The retry delay had no jitter.

Requests that failed at approximately the same time waited for approximately the same duration and retried at approximately the same time. The first wave of traffic created a second wave.

A fixed 200-millisecond delay behaves like this:

request A: fail ── 200 ms ── retry
request B: fail ── 200 ms ── retry
request C: fail ── 200 ms ── retry

Exponential backoff with jitter spreads those attempts over a range:

request A: fail ── 143 ms ── retry
request B: fail ───── 267 ms ───── retry
request C: fail ───────── 391 ms ───────── retry

Backoff reduces the rate at which a client returns to a failing dependency. Jitter reduces synchronization between clients.

Neither creates capacity. If the dependency remains slow, a delayed retry can still be wasted work.

The load test exercised only the healthy path

The original test proved one useful fact: the healthy happy path could handle the expected concurrency.

It did not prove that the service could survive dependency degradation.

We had not tested:

• Four-second downstream latency
• Enough timeouts or retryable responses to activate every allowed attempt
• Retry synchronization
• HTTP connection-pool waiting
• Tomcat thread occupancy during slow calls
• Exhaustion of the retry budget
• Bulkhead rejection behavior
• Recovery after the dependency became healthy

If a production service permits retries, a representative load test must activate them. Otherwise, it tests only a busier good day.

The fix was not a different retry count

We did not solve the design problem by changing three attempts to another arbitrary number.

Retries had to share the same deadlines and capacity controls as the rest of the request.

The controls were:

• Retry only failures likely to be transient.
• Use exponential backoff with jitter.
• Keep connection waiting, execution, backoff, and every attempt inside one customer deadline.
• Do not start an attempt when too little useful time remains.
• Limit retry traffic with a retry budget.
• Bound downstream concurrency separately from the full Tomcat thread pool.
• Reject excess work before every thread and connection is occupied.
• Measure every outbound attempt, not only customer requests.

The trade-off was deliberate. Some requests would fail quickly during degradation. In return, unrelated endpoints remained responsive and the application retained enough capacity to recover.

A controlled failure is often safer than a large collection of slow timeouts.

An illustrative Spring Boot implementation

The incident record does not identify the original retry library, timeout values, pool sizes, retry-budget rule, or bulkhead limit. The following example is therefore a reference design, not code copied from the production service.

It uses Spring Boot 3.5.7, OkHttp 4.9.2, Resilience4j, Micrometer, WireMock, and k6. In this example, three attempts means one original call plus at most two retries.

The example search operation is a read-only GET, so repeating it is safe. A write operation needs a separate idempotency design, usually including an idempotency key understood by the downstream service.

Dependencies

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("com.squareup.okhttp3:okhttp:4.9.2")
    implementation("io.github.resilience4j:resilience4j-retry:2.3.0")
    implementation("io.github.resilience4j:resilience4j-bulkhead:2.3.0")
    implementation("io.micrometer:micrometer-registry-prometheus")

    testImplementation("org.wiremock:wiremock-standalone:3.13.1")
}

Verify the latest compatible Resilience4j and WireMock patch versions against your project dependency graph. The explicit versions above make the example reproducible; they do not imply that the incident used those versions.

Make the HTTP-client limits explicit

@Bean
OkHttpClient downstreamClient() {
    Dispatcher dispatcher = new Dispatcher();
    dispatcher.setMaxRequests(64);
    dispatcher.setMaxRequestsPerHost(32);

    return new OkHttpClient.Builder()
        .dispatcher(dispatcher)
        .connectTimeout(Duration.ofMillis(250))
        .readTimeout(Duration.ofMillis(1200))
        .callTimeout(Duration.ofMillis(1500))
        .retryOnConnectionFailure(false)
        .build();
}

callTimeout includes DNS, connection establishment, connection-pool waiting, request writing, server processing, and response reading for one OkHttp call. The outer customer deadline in the next section includes all attempts and backoff.

The example disables OkHttp’s automatic connection retry so the Resilience4j policy is the only retry owner. Another valid design may keep it enabled, but those hidden attempts must then be included in the capacity model and metrics.

The dispatcher limits are not the connection-pool size. They bound running asynchronous calls. For synchronous execute() calls, the application bulkhead below is the important concurrency control. The connection pool mainly controls reusable idle connections; it is not a substitute for a bulkhead.

Configure retry and bulkhead as code

@Bean
Retry searchRetry() {
    IntervalFunction delay = IntervalFunction.ofExponentialRandomBackoff(
        Duration.ofMillis(100), // initial delay
        2.0,                    // exponential multiplier
        0.5                     // randomization factor
    );

    RetryConfig config = RetryConfig.custom()
        .maxAttempts(3) // original + at most two retries
        .intervalFunction(delay)
        .retryExceptions(
            SocketTimeoutException.class,
            ConnectException.class,
            RetryableDownstreamException.class
        )
        .ignoreExceptions(
            InvalidSearchRequestException.class,
            RequestDeadlineExceededException.class
        )
        .build();

    return Retry.of("search-downstream", config);
}

@Bean
Bulkhead searchBulkhead() {
    BulkheadConfig config = BulkheadConfig.custom()
        .maxConcurrentCalls(24)
        .maxWaitDuration(Duration.ZERO)
        .build();

    return Bulkhead.of("search-downstream", config);
}

Only connection failures, socket timeouts, and explicitly translated transient responses are retryable. In this scenario, 429, 502, 503, and 504 can be translated to RetryableDownstreamException; most 4xx responses should not be retried.

The zero wait time makes the bulkhead reject immediately when its 24 permits are occupied. That protects the rest of the application from waiting behind work that has not started. Twenty-four is only an example: a production limit should come from measured downstream latency, arrival rate, instance count, connection capacity, and the amount of concurrency the dependency can safely accept.

Put every attempt inside the customer deadline

SearchResult search(String query, Duration customerTimeout) {
    Deadline deadline = Deadline.after(customerTimeout); // 2.5 s in this example
    AtomicInteger attempt = new AtomicInteger();

    Supplier<SearchResult> oneAttempt = () -> {
        int number = attempt.incrementAndGet();
        Duration remaining = deadline.remaining();

        if (remaining.compareTo(Duration.ofMillis(500)) < 0) {
            throw new RequestDeadlineExceededException();
        }

        return bulkhead.executeSupplier(() ->
            executeSearch(query, number, remaining)
        );
    };

    return retry.executeSupplier(oneAttempt);
}

Deadline is a small application helper built on a monotonic clock; it is not a JDK or Resilience4j class. executeSearch caps OkHttp’s per-call timeout to remaining and translates only the selected transient HTTP responses into RetryableDownstreamException.

The important decision is the decorator order. The retry policy invokes oneAttempt, and every attempt must acquire a bulkhead permit. A retry cannot bypass the downstream concurrency limit.

Before each attempt, the code checks the remaining end-to-end deadline. If less than 500 milliseconds remains, it stops. The per-call timeout should also be capped to the remaining deadline inside executeSearch; otherwise, an individual attempt can outlive the customer request.

Scheduled backoff must also fit inside the deadline. A production implementation can use a deadline-aware interval function or a retry event hook to prevent sleeping past the point where another useful attempt is possible.

Apply a retry budget

A bulkhead limits concurrent downstream work. A retry budget limits how much of that work may be retries.

For example, allow no more than 10 retry attempts for every 100 original attempts over a rolling window. When the budget is empty, the original call may proceed, but another retry does not.

if (attemptNumber > 1 && !retryBudget.tryAcquire()) {
    throw new RetryBudgetExhaustedException();
}

The exact ratio is a capacity decision, not a universal recommendation. Ten percent may be too high for an expensive dependency and too low for a service with rare, short connection failures.

The budget can use a token bucket local to each instance, or a centrally coordinated mechanism when strict fleet-wide control is required. A local budget is simpler and avoids turning retry admission into another remote dependency, but scaling the application changes the fleet-wide retry allowance.

Record attempts separately

void recordAttempt(int attempt, String outcome) {
    Counter.builder("client.request.attempts")
        .tag("client", "search-downstream")
        .tag("request.type", attempt == 1 ? "original" : "retry")
        .tag("attempt.number", Integer.toString(attempt))
        .tag("outcome", outcome)
        .register(meterRegistry)
        .increment();
}

Useful outcomes include success, timeout, rejected, and error. These tags have small, bounded value sets.

Do not add customer IDs, query text, trace IDs, or request IDs as metric labels. Their unbounded cardinality can make the monitoring system expensive or unstable. Those values belong in sampled logs or traces.

Build a failure-mode load test

The next test needs a controllable downstream dependency. WireMock is sufficient for application-level latency and HTTP failures. Toxiproxy is useful when the test must model connection establishment, bandwidth, resets, or other network behavior.

Inject 200-millisecond and four-second responses

Start WireMock on port 9090, then change its stub between test phases.

Healthy mapping:

{
  "request": { "method": "GET", "urlPath": "/search" },
  "response": {
    "status": 200,
    "fixedDelayMilliseconds": 200,
    "jsonBody": { "items": [] },
    "headers": { "Content-Type": "application/json" }
  }
}

Degraded mapping:

{
  "request": { "method": "GET", "urlPath": "/search" },
  "response": {
    "status": 200,
    "fixedDelayMilliseconds": 4000,
    "jsonBody": { "items": [] },
    "headers": { "Content-Type": "application/json" }
  }
}

Because the example client has a 1.5-second call timeout, the four-second stub activates the timeout and retry path. Add a separate scenario that returns intermittent 503 responses to test status-based retries.

Do not point destructive fault injection at a shared production dependency. Run it in an isolated performance environment with realistic application and pool configuration.

Keep incoming traffic steady with k6

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    search: {
      executor: 'constant-arrival-rate',
      rate: 10,
      timeUnit: '1s',
      duration: '9m',
      preAllocatedVUs: 100,
      maxVUs: 300,
    },
  },
};

export default function () {
  const response = http.get('http://localhost:8080/api/search?q=coffee', {
    timeout: '3s',
  });

  check(response, {
    'success or controlled rejection': (r) =>
      r.status === 200 || r.status === 503,
  });
}

Use a constant arrival rate rather than a closed loop that waits before sending the next request. A closed-loop test silently reduces offered load when the service becomes slow and can hide the overload behavior being investigated.

Run three three-minute phases while k6 continues at the same rate:

0–3 min: Baseline at 200 ms
3–6 min: Dependency degradation at 4 s, plus intermittent 503s
6–9 min: Recovery at 200 ms

Switch WireMock scenarios through its admin API, or use a small test controller available only in the isolated test environment.

What the test should prove

The practical sequence is:

1. Establish a baseline at approximately 200 milliseconds.
2. Increase downstream latency to four seconds.
3. Inject a small percentage of retryable failures.
4. Maintain a steady incoming request rate.
5. Confirm that the retry path activates.
6. Compare incoming requests with downstream attempts.
7. Exhaust the retry budget intentionally.
8. Verify immediate bulkhead rejection and the API’s controlled 503 response.
9. Restore the dependency and measure recovery time.
10. Repeat with latency ramps, bursts, connection failures, and different error ratios.

One successful run is not enough. Retry behavior depends on timing, so repeated runs should produce a bounded range rather than one lucky result.

Put these metrics on one dashboard

The dashboard should make amplification and capacity visible together:

• Incoming request rate
• Downstream attempt rate
• Attempts per customer request
• Original calls versus retries
• Retry success rate
• Retry-budget utilization and denial count
• Tomcat current and busy threads
• Downstream calls running and waiting
• Bulkhead utilization and rejections
• End-to-end latency percentiles
• Downstream attempt latency
• Timeout and controlled-rejection counts
• Recovery time after the dependency becomes healthy

For OkHttp synchronous calls, instrument the call lifecycle with an EventListener or an application interceptor. Do not assume dispatcher queue metrics describe synchronous execute() traffic. Measure connection acquisition and total call time directly.

The most useful graph overlays incoming request rate, original downstream calls, retries, and bulkhead rejections. A retry storm becomes visible immediately instead of being hidden inside one total HTTP-client counter.

Before and after: what can honestly be reported

The incident gives us the following measured before-fix values. The post-fix numbers were not supplied, so they should be filled from the reproducible test rather than invented.

This is more useful than claiming that retries improved the success rate. A retry is justified only when measured data shows that its additional success value exceeds its capacity cost.

After the controls are added, a passing result should show:

• Retry amplification remains below the selected budget.
• Jitter spreads retries instead of producing obvious waves.
• The bulkhead prevents slow calls from consuming every Tomcat thread.
• Requests with insufficient remaining deadline do not start another attempt.
• Some overload is returned quickly as a controlled 503 response.
• Unrelated endpoints remain responsive.
• The application recovers shortly after downstream latency returns to normal, without draining a large retry backlog first.

These are acceptance criteria. Replace them with measured values before presenting the test as evidence.

Important production notes

A timeout does not guarantee cancellation downstream

When the client stops waiting, the downstream operation may continue. Retrying can create overlapping work even though the first attempt looks finished from the caller’s perspective.

This is especially dangerous for writes. Use idempotency keys and confirm the dependency’s cancellation semantics before enabling retries.

One deadline must cover all waiting

The customer deadline includes more than socket read time. It must cover bulkhead or executor waiting, connection-pool acquisition, connection establishment, execution, response reading, backoff, and every attempt.

Independent timeouts can easily add up to a response time longer than the customer is willing to accept.

Bulkhead sizing is an operational decision

A limit that is too high delays exhaustion rather than preventing it. A limit that is too low rejects useful traffic during small latency fluctuations.

Use load-test evidence, per-instance capacity, and downstream service limits. Then monitor rejections in production and revise the limit deliberately.

Memory still matters

Every waiting request retains request state, stack frames, buffers, trace context, and other objects. A retry design that avoids high CPU can still create memory pressure and long garbage-collection pauses through excessive concurrency and queues.

Prefer bounded queues and immediate admission decisions.

Retryable does not mean always retry

429 may include a Retry-After instruction. A 503 may last for minutes. A timeout may mean the dependency is still doing the work. The error type, remaining deadline, retry budget, operation safety, and current capacity must all permit another attempt.

Engineering lessons

• Retry traffic belongs in the capacity model. Incoming request rate does not equal downstream call rate.
• A healthy-dependency load test cannot prove resilience during degradation.
• Backoff without jitter can still create synchronized traffic, and jitter does not create capacity.
• A retry should spend a limited deadline budget and a limited capacity budget.
• Controlled rejection is safer than allowing every request to become a slow timeout.

What we changed—and what the test must prove

Our original load test proved that the API could handle expected traffic while its dependency responded in 200 milliseconds. It did not test the failure mode that mattered.

Production asked a different question.

If the dependency becomes 20 times slower, will 300 customer requests still produce only 300 downstream calls?

In our case, 300 customer requests produced nearly 900 downstream attempts.

We rebuilt the test around a controllable slow dependency, steady arrival rate, attempt-level metrics, retry-budget exhaustion, bulkhead rejection, and recovery. We also put every attempt and backoff period inside one customer deadline.

The lesson was simple: retries are part of the production workload. They must be part of the load test too.