Our Redis Lock Expired. The Spring Boot Job Kept Running.

Why a ten-minute Redis lease allowed overlapping report jobs, and what we need to test before choosing a fix.

Two Spring Boot pods were running the same report at the same time, increasing system load. Both had successfully acquired the Redis lock.

They acquired it at different times. The first report ran longer than the lock’s ten-minute time to live (TTL). After it expired, another pod acquired the lock while the original worker kept going.

That was the boundary we had missed: a Redis lock with a TTL is a lease. Its expiry makes the lock available again. It does not stop the Java code that acquired it.

We have not established that users received duplicate reports or that results were corrupted. The confirmed impact was increased load.

The fix is still pending. This article covers the incident, a proposed local reproduction, and the safeguards we need to test. The reproduction has not been run, and there are no measured production improvements to report.

We assumed the lock covered the whole job

The assumption was easy to make: if the worker acquired the distributed lock, it could process the report exclusively.

That assumption needed a time boundary.

Successful acquisition described ownership at the start. The report kept running after that ownership expired. Redis’s distributed locking guidance explains this limited validity window.

Redis did not need to grant both pods a valid lease simultaneously. The leases belonged to different periods; the executions overlapped.

The first worker did not need to crash. It only needed to run for longer than ten minutes.

Tracing the same job across two pods

When duplicate work appears in a system using Kafka, message delivery is a reasonable place to investigate.

In this case, we found no duplicate Kafka messages explaining the overlap. The logs showed different pods processing the same job.

Those findings answer different questions. Delivery evidence helps explain how work arrived. Execution evidence shows what the workers actually did. Neither should substitute for the other.

We still need to establish what caused the second pod to attempt the same job. Scheduling, retry, polling, or another path could explain it. Lease expiry explains why the attempt could acquire the lock; it does not identify the trigger.

To trace the next overlap, we need to correlate:

• jobId: the stable identity of the logical report request.
• executionId: a new identifier for each processing attempt.
• pod: the worker instance handling that attempt.
• Lock acquisition, job start, renewal outcome, ownership loss, and job finish events.
• Kafka topic, partition, offset, and application message identity where the trigger involves Kafka.

Two attempts need different execution IDs and the same job ID if they represent the same report request. The lock key must reflect that same logical identity. Whether it includes a tenant, reporting period, input snapshot, or request version depends on the application.

Here is an illustrative logging format, not a production log excerpt:

{"jobId":"report-42","executionId":"attempt-a","pod":"pod-a","event":"lease_renewal","elapsedMs":4200,"renewalOutcome":"timeout"}
{"jobId":"report-42","executionId":"attempt-a","pod":"pod-a","event":"job_stop_requested","elapsedMs":4201,"reason":"ownership_uncertain"}

These fields connect a lease event to a specific execution. A stop request should also have a later event showing whether the worker actually stopped and how long that took.

Before choosing production settings, we still need exact timestamps, runtime percentiles, load measurements, and the locking library and Redis topology. Those details are not available here.

The lease expired before the report finished

The available facts support this relative timeline:

The key transition is lease expiry: A remains active while B becomes the new owner.

It also gives us a small failure to reproduce: let one job remain active beyond its TTL, then have another worker acquire the same key.

A local reproduction of the overlap

The following demonstration uses two Spring Boot processes against one local Redis instance. It deliberately contains the lifecycle mistake: worker A continues after its lease expires.

It uses a five-second demo TTL, not the production ten-minute setting. Coordination keys keep A’s simulated job open until B starts, so the demonstration does not depend on launching two terminals at exactly the right moment.

This deliberately incomplete worker is intended to demonstrate overlapping execution. Its waits do not reproduce the report’s CPU, database, or memory load, and it is not a production locking implementation.

1. Prepare the application

Use a Maven Spring Boot 3.5.x project with Java 21 and Spring Data Redis. Keep the Redis dependency version managed by the project’s Spring Boot parent or BOM:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

This dependency supplies the Redis integration used by StringRedisTemplate. The project also needs the usual Spring Boot Maven plugin to package an executable JAR.

Start a disposable Redis 7.4 instance with Docker:

docker run --rm --name lease-demo-redis -p 127.0.0.1:6379:6379 redis:7.4

Keep that terminal open. The loopback binding is for local development. If port 6379 is occupied, map an unused host port to the container’s 6379 and set spring.data.redis.port below to that host port.

Put this in src/main/resources/application.yml:

spring:
  main:
    web-application-type: none
  data:
    redis:
      host: localhost
      port: 6379
      connect-timeout: 1s
      timeout: 1s

These are demo connection and command timeouts. Spring Boot documents both properties in its application property reference.

2. Add a worker that outlives its lease

Replace the generated application class with src/main/java/example/LeaseDemoApplication.java. Remove the old application class so this standalone demo has a single main entry point. If a generated context test references the old class or package, update it to reference example.LeaseDemoApplication.

package example;

import java.time.Duration;
import java.util.List;
import java.util.UUID;
import java.util.function.BooleanSupplier;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;

@SpringBootApplication
public class LeaseDemoApplication {
    private static final Duration LEASE = Duration.ofSeconds(5);
    private static final Duration MARKER_TTL = Duration.ofMinutes(2);
    private static final DefaultRedisScript<Long> RELEASE =
        new DefaultRedisScript<>("""
            if redis.call('GET', KEYS[1]) == ARGV[1] then
                return redis.call('DEL', KEYS[1])
            end
            return 0
            """, Long.class);

    public static void main(String[] args) throws Exception {
        if (args.length != 2 || !List.of("A", "B").contains(args[0])) {
            throw new IllegalArgumentException("Use: A|B unique-run-id");
        }
        try (var context = SpringApplication.run(
                LeaseDemoApplication.class, args)) {
            run(context.getBean(StringRedisTemplate.class), args[0], args[1]);
        }
    }

    private static void run(StringRedisTemplate redis, String role,
                            String runId) throws Exception {
        var values = redis.opsForValue();
        String base = "lease-demo:" + runId;
        String key = base + ":report-42:lock";
        String owner = UUID.randomUUID().toString();
        long started = System.nanoTime();

        if (role.equals("B")) {
            await(() -> Boolean.TRUE.equals(redis.hasKey(base + ":a-started")));
        }

        await(() -> Boolean.TRUE.equals(
            values.setIfAbsent(key, owner, LEASE)));

        try {
            log(role, owner, "acquired", started);
            if (role.equals("A")) {
                values.set(base + ":a-started", "1", MARKER_TTL);
                // Intentionally keep A's simulated job open beyond its TTL.
                await(() -> Boolean.TRUE.equals(
                    redis.hasKey(base + ":b-started")));
                log(role, owner, "still_running_after_b_started", started);
            } else {
                values.set(base + ":b-started", "1", MARKER_TTL);
                // B stays active until A has attempted its late release.
                await(() -> Boolean.TRUE.equals(
                    redis.hasKey(base + ":a-finished")));
                log(role, owner, "finished", started);
            }
        } finally {
            Long deleted = redis.execute(RELEASE, List.of(key), owner);
            log(role, owner, "release_deleted_" + deleted, started);
            if (role.equals("A")) {
                values.set(base + ":a-finished", "1", MARKER_TTL);
            }
        }
    }

    private static void await(BooleanSupplier condition) throws Exception {
        long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos();
        while (!condition.getAsBoolean()) {
            if (System.nanoTime() >= deadline) {
                throw new IllegalStateException("Demo coordination timed out");
            }
            Thread.sleep(100);
        }
    }

    private static void log(String pod, String executionId, String event,
                            long started) {
        long elapsedMs = (System.nanoTime() - started) / 1_000_000;
        System.out.printf(
            "jobId=report-42 executionId=%s pod=%s event=%s elapsedMs=%d%n",
            executionId, pod, event, elapsedMs);
    }
}

Worker B waits for A’s start marker, then retries acquisition until the lease becomes available. A keeps its job active until B’s start marker appears. The handshake makes the overlap observable, and the waits have a timeout so a missing peer fails the demonstration.

The unique ownership value and conditional release protect B’s lock from A’s late cleanup. They do not prevent A from continuing its work. Spring Data Redis supports this Lua execution through the template’s scripting API.

3. Run two instances

Package the project, then start A in one terminal:

./mvnw clean package
java -jar target/lease-demo-0.0.1-SNAPSHOT.jar A run-001

Use the JAR name produced by your project. On Windows, the Maven wrapper command is .\mvnw.cmd clean package.

While A is waiting, start B in another terminal:

java -jar target/lease-demo-0.0.1-SNAPSHOT.jar B run-001

Use the same run ID in both terminals, and a fresh ID for every repeat. Start B within A’s 60-second coordination window. B polls every 100 milliseconds for this local test; a real contention strategy needs bounded retries and backoff.

The expected event order, not captured test output, is:

A: acquired
B: acquired                         # after A's five-second lease expires
A: still_running_after_b_started
A: release_deleted_0                # A cannot remove B's lease
B: finished
B: release_deleted_1                # if B's own lease is still valid

The important result is B acquiring before A finishes. A’s release should return zero because its ownership value no longer matches. Long process pauses can also cause B’s lease to expire before cleanup, in which case B’s release returns zero too.

Each process measures elapsedMs from its own start inside run, so those values are not a shared timeline. The marker handshake establishes the event order. For incident analysis, include wall-clock timestamps and trace context alongside local durations.

The demo also leaves cleanup failure handling minimal: if release throws, it can hide the original failure and prevent A’s finish marker from being written. A production worker must preserve the original outcome and record cleanup failures separately.

Stop the Redis container with Ctrl+C when finished. The coordination keys also have their own expiry.

Decide what should happen when ownership is lost

Before changing the TTL, we need two operational decisions: how long a report may run, and how long another worker may wait to take over after a crash.

A proposed experiment might use the following settings. The report properties are custom application settings; the demo above does not bind or implement them.

report:
  lease-duration: 30s
  renewal-interval: 5s
  max-runtime: 20m

spring:
  data:
    redis:
      connect-timeout: 1s
      timeout: 1s

These are trial values, not recommended production defaults. Renewal needs enough time for scheduling delay and Redis latency before the current lease expires. Maximum runtime bounds the job even if renewal keeps succeeding.

We need to measure job runtime, renewal latency, worker pauses, and stopping delay before choosing those values. A renewal interval that looks comfortable during normal operation may fail when the process is paused or its executor is saturated.

Here is the proposed lifecycle in pseudocode:

create a unique ownership value for this acquisition attempt
attempt acquisition with a bounded Redis timeout
if acquisition is denied or its outcome is uncertain: do not start work

establish a conservative local lease deadline
start a renewal task independent of the report's work loop

renewal task:
    renew only while the current local lease is still considered valid
    on confirmed, timely success: advance the local deadline
    on rejection, timeout, exception, or late response:
        permanently mark this execution as ownership-uncertain
        request cancellation; do not revive this attempt

before each bounded work unit and completion attempt:
    stop if cancelled, ownership-uncertain, or past either deadline
    perform work with its own bounded I/O timeouts

complete through the destination's durable duplicate/stale-write policy

finally:
    stop and coordinate with the renewal task
    attempt ownership-checked release with a bounded timeout
    record cleanup failure without hiding the original job outcome

The local deadline should account for time spent acquiring or renewing, using a monotonic elapsed-time source and a conservative margin. A success response that arrives too late cannot justify another full lease interval starting at response time.

Cancellation is cooperative. A checkpoint can stop the next batch, but it cannot recall a request already sent to another service. A paused worker may resume after another worker starts. Those boundaries remain relevant even with renewal, as discussed in Martin Kleppmann’s analysis of distributed locking.

What each safeguard can solve

Each safeguard addresses a different part of the failure:

Renew and release only the matching owner

For the Redis 7.4 demo, a renewal operation can use this Lua script:

-- KEYS[1]: lock key
-- ARGV[1]: ownership value
-- ARGV[2]: positive lease duration in milliseconds
if redis.call('GET', KEYS[1]) == ARGV[1] then
    return redis.call('PEXPIRE', KEYS[1], ARGV[2])
end
return 0

The ownership comparison and expiry update execute atomically. A zero result means renewal did not happen; a client timeout leaves the outcome uncertain. The caller must handle both deliberately. The command’s millisecond argument is documented under PEXPIRE.

Release needs the same atomic ownership comparison, as in the Java example. A separate GET followed by DEL leaves a race. These scripts follow the ownership checks in Redis’s locking guidance.

The example uses one Redis instance. Production review also needs the actual library, version, failover behavior, and deployment topology; the same guidance describes how asynchronous replication can lose a lock during failover. TTL testing alone does not validate that setup.

Protect the durable result

We have not confirmed the report’s persistence backend or external side effects, so a PostgreSQL-specific fix would be premature.

The requirement can still be stated: recording the completion identity and applying its associated database effect must form an atomic, durable operation. A separate “already completed?” read followed by a write can race. The AWS Builders’ Library discussion of idempotent APIs explains why the identity and mutations belong together.

For this report, the design must define what a second completion receives: the existing result, a conflict, or another explicit outcome. The identity also needs to distinguish a retry from an intentional new report using different inputs.

File uploads, notifications, and downstream calls need their own handling. A database transaction cannot automatically make those remote effects atomic. For example, a notification path may need a durable outbox plus an idempotent receiver.

Even if only one result is accepted, two workers may still perform the expensive calculation. Since increased load was the observed impact, we must measure that wasted work separately.

Consider fencing if stale writes threaten correctness

A Redis ownership check immediately before a database write leaves a gap. The worker can pause after the check, lose its lease, and later send the write.

Fencing uses an increasing ownership sequence. The destination atomically rejects a token older than the highest it has accepted. A random UUID identifies an owner but supplies no ordering. Token allocation must preserve that ordering across failures, and the destination must enforce it; attaching a number alone does nothing. Kleppmann’s fencing explanation describes this responsibility.

Whether this report needs fencing depends on its write semantics. We should establish those before adding another mechanism.

Test the failure paths

The next tests should coordinate events explicitly, just as the reproduction coordinates A and B. Exact sleeps alone make concurrency tests fragile.

For the late-operation test, inspect B’s value and remaining TTL while its lease is still valid. For completion testing, use the real persistence boundary. Counting successful lock acquisitions cannot prove result correctness.

How we will judge a fix

These are validation goals, not achieved results:

• The reproduction demonstrates that lease expiry permits B to start while A continues.
• An old worker cannot release or renew a newer worker’s lease.
• Ownership loss produces deliberate, observable behavior at defined stopping points.
• Repeated completion attempts have a defined durable outcome.
• Measurements distinguish overlapping computation from duplicate committed effects.
• Recovery after a worker crash meets the agreed takeover target.

For load, record job runtime distributions, overlapping execution duration, and resource consumption during overlap. For effects, record accepted, duplicate, and rejected completion attempts separately.

Keep job and execution IDs in logs or traces. Using every ID as a metric label would create a separate cardinality problem.

What we learned

• Successful lock acquisition does not guarantee exclusivity for the full job. The worker must account for lease expiry during execution.
• Ownership checks protect renewal and release. They do not stop stale application work.
• Renewal and cancellation need explicit failure behavior, including what happens after a process pause or an uncertain Redis response.
• Preventing duplicate committed results and reducing duplicate computation are separate goals. Increased load was the impact we observed.

The next step is to test the boundary

We traced the overlap to a report outliving its ten-minute lease. The original worker continued after another pod acquired the lock.

That explains why the executions overlapped. We still need to identify the second attempt’s trigger and test a fix against our runtime, recovery, and correctness requirements.

The proposed reproduction gives us a way to exercise the failure. Next, we need to define stopping and completion behavior, then test renewal against those requirements. A longer TTL may help, but we still need an answer for what happens when it expires.

The review question I want to keep asking is:

If this lease expires halfway through the job, how does the worker find out, and what happens next?