Why Our Spring Boot Health Check Lied During an Outage

Editorial disclosure: This is a synthetic case study based on the approved CS-01 scenario. It does not describe a verified outage at a real organization. All incident details and test results are constructed.

A Spring Boot health check can return UP while the application is unavailable to customers.

The apparent contradiction usually means that the health endpoint and the business request measure different execution paths. A probe may verify process state and database connectivity through a dedicated management port while customer requests compete for an exhausted application thread pool and wait on a slow downstream service.

That is what happened in this synthetic checkout incident.

For 23 minutes, 38% of checkout attempts timed out. Every Kubernetes pod remained ready. Spring Boot Actuator continued responding in 8–14 milliseconds.

The readiness result was technically accurate. The operational interpretation was not.

The team had treated UP as evidence that a pod could complete checkout. The configured probe only established that Spring considered the process ready and that PostgreSQL was reachable through the management path.

System Context

The checkout API ran Spring Boot 3.2.5 in Kubernetes. Customer traffic entered through port 8080, while Spring Boot Actuator ran on a separate management port, 9091.

Kubernetes used this endpoint as its readiness probe:

GET :9091/actuator/health/readiness

The relevant configuration was:

server:
  port: 8080
  tomcat:
    threads:
      max: 200

management:
  server:
    port: 9091
  endpoint:
    health:
      group:
        readiness:
          include: readinessState,db

Checkout called a pricing service synchronously before confirming an order. Under normal conditions, pricing responded in approximately 80 milliseconds.

The pricing client had a 15-second read timeout. The ingress stopped waiting for checkout after five seconds. Those settings appeared harmless while pricing was fast. Their relationship became destructive when the dependency slowed down.

Incident Symptoms

At 10:02, pricing latency began increasing. By 10:04, checkout timeout alerts were firing. At 10:06, the on-call engineer saw every checkout pod marked ready and began investigating the ingress.

The remaining observations appeared reassuring:

• CPU stayed below 55%.
• PostgreSQL connection-pool utilization stayed below 35%.
• Database query latency did not materially change.
• Actuator returned UP in 8–14 milliseconds.

Meanwhile, pricing latency had risen to approximately 12 seconds, all 200 Tomcat request threads had become occupied, and 38% of checkout attempts were timing out at the ingress.

The scenario contains no evidence of lost or duplicated orders.

The combination of green pods and customer failures directed the investigation toward the ingress and Kubernetes routing. That was a reasonable starting point, but it depended on an unstated assumption: a ready pod must have enough capacity to complete checkout.

The readiness configuration made no such promise.

How the Investigation Narrowed the Failure

Requests Were Reaching the Application

The ingress was the first suspected component. If every destination was ready but customers were timing out, traffic might have been lost or misrouted between the load balancer and the pods.

Ingress metrics showed a different pattern. The ingress accepted checkout requests, waited for five seconds, and then timed out. Matching distributed traces appeared inside the checkout application.

This evidence rejected two early theories: the ingress was not dropping requests before they reached Kubernetes, and Kubernetes was not routing requests to the wrong destination.

Traffic reached the expected checkout pods. Those pods simply failed to complete the work before the external deadline. The investigation therefore shifted from connectivity to execution capacity.

PostgreSQL Was Healthy but Checkout Was Not

PostgreSQL was the next plausible bottleneck. A slow query or exhausted connection pool could have held application threads long enough to explain the timeouts.

The database metrics did not support that theory. Connection-pool utilization remained below 35%, and query latency did not materially change. The database health indicator was green because PostgreSQL was genuinely reachable.

A healthy database is evidence about the database path. It does not prove that every other dependency involved in checkout is healthy. It also does not establish that the application has enough request-processing capacity to use the database on behalf of a customer.

The database signal was correct but incomplete.

Low CPU Concealed Saturation

CPU remained below 55%. That initially seemed inconsistent with an overloaded service. In many capacity incidents, CPU saturation is the most visible symptom.

Java services can become unavailable without consuming all available CPU. A request thread waiting on remote I/O performs little computation, but it remains unavailable to other requests.

Tomcat’s active-thread metric revealed the actual boundary:

active request threads: 200
maximum request threads: 200

The application had exhausted its request pool.

New traffic could still reach the pod. The process could still answer lightweight requests through a separate server. The JVM could still have idle CPU. None of those facts created a free request thread on port 8080.

Traces Identified the Slow Dependency

Distributed traces attributed most checkout latency to the synchronous pricing call.

Pricing latency had increased from approximately 80 milliseconds to approximately 12 seconds. Every affected checkout request held an application thread while waiting for that dependency.

A thread dump at 10:14 confirmed the trace evidence. Most application request threads were waiting for pricing responses.

That explained how the request pool became exhausted. It still left one question: why could the readiness endpoint respond in milliseconds while customer traffic could not progress?

The Probe Used a Different Execution Path

At 10:17, the team compared the paths directly.

The customer path entered through port 8080, acquired a Tomcat application thread, executed checkout logic, and waited synchronously for pricing.

The readiness path entered through port 9091, used the management server, and checked only Spring’s readiness state and PostgreSQL.

The probe did not call pricing. It did not execute checkout. It did not measure free Tomcat threads on port 8080.

Because Actuator used a separate management server, it could remain responsive while the customer-facing request pool was exhausted.

This was not merely a missing health indicator. It was a mismatch between the system operators thought they were observing and the path the probe actually exercised.

Root Cause

The outage mechanism had three connected parts.

Slow Pricing Calls Occupied Every Request Thread

Pricing latency increased to approximately 12 seconds. Checkout used a blocking, synchronous call, so every in-flight request retained a Tomcat thread while waiting.

Those waits accumulated until all 200 application request threads were occupied. The repository does not establish the exact incoming production request rate, so no unsupported concurrency calculation should be inferred. The thread metric and thread dump establish the saturation mechanism within this synthetic scenario.

The Timeout Hierarchy Preserved Abandoned Work

The ingress timeout was five seconds. The pricing read timeout was 15 seconds.

After five seconds, the ingress returned a timeout to the caller. Checkout could nevertheless continue waiting for pricing for up to another ten seconds.

The external request had ended, but the internal work had not. Those abandoned operations retained application capacity without any possibility of returning a useful response to the original caller.

Readiness Had a Narrower Contract

The readiness group checked readinessState and db. During the incident, Spring’s readiness state remained valid, PostgreSQL remained reachable, and the management server remained responsive.

The readiness endpoint returned UP because every configured check succeeded.

The operational mistake was treating that result as if it combined process state, routing suitability, downstream availability, request capacity, and customer success.

One endpoint did not answer all of those questions.

Why Pricing Was Not Added to Readiness

Adding pricing to the readiness group was the most obvious proposed correction. It would also have introduced a fleet-level failure mode.

Every checkout pod shared the same pricing dependency. If pricing failed globally, every pod could report unready at approximately the same time. Kubernetes could then remove the entire checkout fleet from service.

That response might prevent new checkout requests, but it would also eliminate capacity that could serve endpoints unrelated to pricing, return a controlled degraded response, expose useful diagnostics, or recover without a fleet-wide readiness transition.

Dependency-aware readiness can be appropriate when a pod has no meaningful behavior without a dependency and removing it from routing reduces harm. It is dangerous when every replica shares the same dependency and will make the same readiness decision.

In this scenario, pricing remained outside readiness. The team instead contained its effect on request capacity.

The Solution

Fit the Pricing Timeout Inside the External Deadline

The pricing connection timeout became 250 milliseconds. The read timeout was reduced from 15 seconds to 1.5 seconds.

pricing-client:
  connect-timeout: 250ms
  read-timeout: 1500ms

This left time inside the five-second ingress deadline for checkout to stop downstream work and return an explicit response.

The values are not universal recommendations. A timeout must reflect the dependency’s latency distribution, the business operation’s deadline, and the time required for local processing and response handling.

The important rule is simple: downstream work must end before the caller stops listening.

Bound Concurrency with a Bulkhead

A bulkhead limited pricing to 50 concurrent calls:

pricing-client:
  bulkhead:
    max-concurrent-calls: 50

When no permit was available, or pricing exceeded its timeout, checkout returned 503 Service Unavailable.

The timeout and bulkhead controlled different dimensions. The timeout limited how long one pricing call could occupy capacity. The bulkhead limited how many pricing calls could occupy capacity simultaneously.

Neither control made pricing available. Together, they prevented one degraded dependency from occupying all 200 request threads.

Add a Customer-Path Signal

A non-mutating synthetic checkout monitor was introduced.

Its purpose was not to decide whether Kubernetes should route traffic. It answered whether the critical customer operation could complete. The monitor had to avoid creating real orders or modifying production data.

Redesign the Dashboard

The dashboard was changed to show application-port thread utilization, pricing latency, bulkhead rejections, Kubernetes readiness, and synthetic checkout status together.

Readiness remained useful, but it stopped functioning as the summary definition of service availability.

Validation Results and Their Limits

The team repeated a controlled load test at 40 checkout requests per second while injecting 12-second pricing latency.

Before the changes, active request threads reached 200 and checkout p95 exceeded the five-second ingress timeout.

After the changes:

• Concurrent pricing calls remained at or below 50.
• Application request threads peaked at 74.
• Failed checkout requests returned with a p95 of 1.7 seconds.
• The synthetic monitor alerted within 60 seconds.

These are synthetic load-test results. They validate resource containment under the specified test conditions.

They do not prove that production reliability improved. The scenario includes no equivalent post-change production incident. A controlled test establishes that the mechanism behaves as designed under test conditions; it does not establish a future production outcome.

Operational Trade-offs and Failure Boundaries

The revised design did not make pricing available.

During pricing degradation, checkout failed earlier and more explicitly. Customers received a controlled 503 instead of waiting for an ingress timeout, while the application preserved capacity for other requests.

That trade-off creates continuing responsibilities:

• Clients need safe retry policies.
• Bulkhead rejection rates require monitoring.
• Timeout values must be reviewed as dependency latency changes.
• The concurrency limit must be revisited as traffic and capacity change.
• The synthetic monitor must remain non-mutating.
• Kubernetes will continue routing traffic during a pricing outage.

The last behavior is deliberate. It is acceptable because the application now protects itself and returns controlled failures. Without those protections, shallow readiness would preserve the original saturation mode.

Production Checklist

• Document the exact routing decision controlled by readiness.
• List every readiness indicator and its failure semantics.
• Determine whether probes and customers share a port, connector, and thread pool.
• Monitor application-port execution capacity.
• Compare downstream timeouts with the external request deadline.
• Terminate work after the caller’s deadline.
• Bound slow dependencies by both time and concurrency.
• Test slow responses, not only refused connections.
• Consider fleet-wide behavior before adding shared dependencies to readiness.
• Measure critical business paths separately.
• Ensure synthetic checks cannot create real transactions.
• Distinguish load-test validation from measured production improvement.
• Review timeout budgets and bulkhead limits as traffic changes.

Engineering Takeaway

A Spring Boot health endpoint is not a universal statement about service availability.

Use readiness for routing decisions. Use resource metrics to detect saturation. Use timeouts and bulkheads to contain downstream failures. Use a customer-path signal to determine whether the system is delivering its actual promise.

Related Spring Boot Production Lessons

• Spring Boot Saturation Test: Why 300 Users Caused 42% Failures
• Resilience4j TimeLimiter in Spring Boot: When Latency Spreads
• Browse Spring Boot production tutorials

Also published on Medium.