How a PostgreSQL-to-Kafka dual write lost business events—and how a transactional outbox made the failure recoverable without pretending to provide exactly-once processing.
Support found the problem before any dashboard did.
Several orders had remained in CREATED far longer than expected. Payment had not started. Inventory had not been reserved. No downstream service appeared to know those orders existed.
PostgreSQL did.
Each order had a valid ID, customer details, line items, and a committed creation timestamp. The Spring Boot request had completed the database write successfully.
Kafka had a different version of the truth: those orders had never happened.
The failure was not in the consumer. It was in the gap between two independent commits.
A transactional outbox closed that gap—not by making PostgreSQL and Kafka one atomic system, but by making failed publication durable, visible, and recoverable.
That distinction became the most important lesson from the incident.
Incident-data note: The original account did not include production counts, versions, or timing values. Every concrete value below is illustrative and is not presented as a measurement from the incident.
We blamed the consumer first
The team began where most Kafka investigations begin: with the consumer.
Perhaps the payment consumer group was lagging. Perhaps deserialization errors had pushed records into a dead-letter topic. Perhaps a deployment had changed offset behavior.
None of those theories survived contact with the evidence.
Consumer lag was normal. There were no matching deserialization failures. The dead-letter topic contained none of the affected order IDs.
The team then searched the order-created topic by record key and payload for those same IDs.
Nothing appeared.
The consumers were not failing to process the events. They had never received them.
The request path looked reasonable at first:
HTTP request
-> validate
-> insert order
-> commit PostgreSQL transaction
-> publish OrderCreated
-> wait for Kafka acknowledgment
The relevant log sequence would look like this. These are illustrative lines, not original incident evidence:
10:41:22.318 INFO OrderService order committed orderId=7fe2... status=CREATED
10:41:32.329 ERROR OrderEvents Kafka acknowledgment timed out orderId=7fe2...
timeoutMs=10000 topic=order-created
PostgreSQL had already made its decision. A Kafka acknowledgment timeout could not undo that commit.
For a concrete running example, assume 37 orders were affected, the oldest had remained in CREATED for 46 minutes, and the service normally handled about 120 order creations per second. Also assume the delivery objective was 99% of events within 10 seconds.
Those values are illustrative. Their purpose is to force operational decisions. A system handling two orders per hour with a five-minute objective can be much simpler than one handling thousands per second with sub-second delivery.
The request succeeded only halfway
The original service effectively did this:
@Transactional
public OrderResponse createOrder(CreateOrderCommand command) {
Order order = orderRepository.save(Order.create(command));
// This call is not part of the PostgreSQL commit.
kafkaTemplate.send("order-created", order.getId().toString(), toEvent(order))
.get(10, TimeUnit.SECONDS);
return OrderResponse.from(order);
}
@Transactional makes the database work transactional. It does not create one atomic commit across an independent PostgreSQL transaction and a Kafka producer.
The exact timing depends on the transaction boundary. The Kafka send might happen before or after the database commit. That changes which inconsistency is possible; it does not remove the dual write.
| PostgreSQL | Kafka | Result |
|---|---|---|
| Commit | Publish succeeds | Consistent |
| Commit | Publish fails | Valid order, missing event |
| Rollback | Publish succeeds | Event for an order that does not exist |
| Rollback | Publish fails | No business change |
Only the first row is the desired result. Two rows create disagreement, and changing the order of two independent calls cannot remove both failure windows.
That was the root cause: the application treated two separate commits as one business operation.
Why the obvious fixes still lose
Publish to Kafka first
This reverses the inconsistency. Kafka can accept OrderCreated, then PostgreSQL can roll back. Payment and inventory may act on an order that does not exist.
Add @Transactional
Spring can coordinate resources in useful ways, but this annotation alone does not turn PostgreSQL and Kafka into a shared atomic resource. It hides neither commit boundary.
Use Kafka producer transactions
Kafka transactions can atomically publish Kafka records and coordinate Kafka consumer offsets. They do not atomically include an independent PostgreSQL commit.
Use Kafka transactions for the guarantee they provide—not as a general substitute for cross-system consistency design.
Retry the HTTP request
A blind retry may create a second order, attempt another charge, or fail on a unique constraint. Even if the API uses an idempotency key and returns the existing order, that does not automatically recreate the missing event.
API idempotency still matters. It solves a different problem: repeated client commands.
Retry Kafka inside the request
An in-request retry can survive a brief network interruption. It cannot make the publication intent durable.
If the process restarts or Kafka remains unavailable beyond the request deadline, the event is still lost. Long retries also occupy request threads, database connections, and memory.
During a broker outage, the application should preserve the work and release the request—not turn the outage into thread exhaustion.
The outbox changed the unit of success
The revised write path used one PostgreSQL transaction:
One PostgreSQL transaction
-> insert order
-> insert outbox event
-> commit both
If the transaction commits, both the order and its publication intent exist. If it rolls back, neither exists.
A separate publisher reads pending outbox rows, sends them to Kafka, and marks them as published after Kafka acknowledges the send.
The outbox does not create a PostgreSQL-to-Kafka transaction. It preserves enough state to recover when either system, the network, or the application process fails.
A focused Spring Boot implementation
This reference implementation uses Java 21, Spring Boot 3.5.16, PostgreSQL 17, and a Kafka 3.9 broker. Spring Boot 3.5.16 supports Java 17 through Java 25 according to its official system requirements.
The versions make the example concrete. Validate them against your dependency policy and broker-client compatibility requirements before using the snippets in a real service.
Dependencies
Spring Boot’s dependency management supplies compatible library versions:
dependencies {
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.springframework.kafka:spring-kafka")
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-registry-prometheus")
implementation("org.postgresql:postgresql")
testImplementation("org.springframework.boot:spring-boot-starter-test")
testImplementation("org.springframework.kafka:spring-kafka-test")
testImplementation("org.testcontainers:postgresql")
testImplementation("org.testcontainers:kafka")
}
The example uses JPA for the business write and native SQL for queue behavior. JDBC is also a good choice for the complete outbox path when a team wants explicit SQL and fewer entity-state surprises.
Store the order and publication intent
The orders table remains part of the business model. The outbox table stores integration state:
create table orders (
id uuid primary key,
customer_id uuid not null,
status varchar(32) not null,
total_amount numeric(19, 2) not null,
event_sequence bigint not null default 0,
created_at timestamptz not null
);
create table outbox_event (
event_id uuid primary key,
aggregate_type varchar(100) not null,
aggregate_id uuid not null,
aggregate_sequence bigint not null,
event_type varchar(150) not null,
payload_version integer not null,
payload jsonb not null,
status varchar(20) not null,
attempts integer not null default 0,
next_attempt_at timestamptz not null,
claimed_by varchar(100),
claimed_until timestamptz,
created_at timestamptz not null,
published_at timestamptz,
last_error varchar(1000),
unique (aggregate_id, aggregate_sequence)
);
create index outbox_ready_idx
on outbox_event (next_attempt_at, created_at)
where status = 'PENDING';
The partial index keeps normal polling focused on pending work. Confirm its usefulness with EXPLAIN (ANALYZE, BUFFERS) at production-like table sizes.
event_id stays stable across retries. aggregate_sequence defines event order within one order. payload_version creates an explicit schema-evolution boundary.
Do not serialize a JPA entity directly into the payload. An entity is an internal persistence model; lazy relationships, field renames, and persistence concerns should not become a long-lived integration contract.
Use a stable event envelope
public record EventEnvelope<T>(
UUID eventId,
String eventType,
UUID aggregateId,
long aggregateSequence,
Instant occurredAt,
int payloadVersion,
T payload
) {}
public record OrderCreatedPayload(
UUID orderId,
UUID customerId,
BigDecimal totalAmount
) {}
The envelope separates routing and deduplication metadata from the business payload. A retry reuses the serialized envelope and the same eventId; it must not create a logically new event.
For JSON contracts, define compatibility rules before changing fields. Avro or Protobuf with schema governance may be a better fit when many teams consume the topic.
Commit both rows together
@Service
public class OrderService {
private final OrderRepository orders;
private final OutboxEventRepository outbox;
private final ObjectMapper objectMapper;
private final Clock clock;
@Transactional
public OrderResponse create(CreateOrderCommand command) {
Instant now = clock.instant();
Order order = Order.create(command, now);
order.advanceEventSequence(); // sequence 1 for OrderCreated
orders.save(order);
UUID eventId = UUID.randomUUID();
EventEnvelope<OrderCreatedPayload> envelope = new EventEnvelope<>(
eventId,
"OrderCreated",
order.getId(),
order.getEventSequence(),
now,
1,
new OrderCreatedPayload(
order.getId(), order.getCustomerId(), order.getTotalAmount()
)
);
outbox.save(OutboxEvent.pending(
eventId,
"Order",
order.getId(),
order.getEventSequence(),
"OrderCreated",
1,
writeJson(envelope),
now
));
return OrderResponse.from(order);
}
}
If JSON serialization fails, the method throws and PostgreSQL rolls back both inserts. If the transaction commits, the publication intent remains durable even if Kafka is unavailable for hours.
The controller can now return after the database transaction. That changes the API’s meaning: the response confirms durable acceptance, not confirmed downstream processing.
If the business requires synchronous payment authorization, an asynchronous outbox alone does not satisfy that requirement.
Claim work without holding locks during Kafka I/O
Multiple publisher instances can claim work with FOR UPDATE SKIP LOCKED. PostgreSQL documents SKIP LOCKED as useful for queue-like access, while warning that it returns an intentionally inconsistent view and should not be used for general-purpose queries.
The publisher should claim a bounded batch in a short database transaction, commit the claims, and call Kafka only after releasing the row locks.
with candidates as (
select e.event_id
from outbox_event e
where e.status = 'PENDING'
and e.next_attempt_at <= now()
and not exists (
select 1
from outbox_event earlier
where earlier.aggregate_id = e.aggregate_id
and earlier.aggregate_sequence < e.aggregate_sequence
and earlier.status <> 'PUBLISHED'
)
order by e.created_at
for update skip locked
limit :batch_size
)
update outbox_event e
set status = 'PROCESSING',
claimed_by = :instance_id,
claimed_until = now() + interval '30 seconds',
attempts = attempts + 1
from candidates c
where e.event_id = c.event_id
returning e.*;
The not exists clause stops sequence 2 from being claimed while sequence 1 for the same order remains unpublished. The lease makes a crashed publisher recoverable.
Test the exact execution plan and concurrency behavior. Queue queries that perform well with thousands of rows can become expensive when millions of published rows remain in the table.
Expose the native query through a small repository implemented with NamedParameterJdbcTemplate:
@Transactional
public List<OutboxEventRow> claimBatch(
String instanceId, int batchSize) {
return jdbc.query(CLAIM_SQL, Map.of(
"instance_id", instanceId,
"batch_size", batchSize
), outboxRowMapper);
}
The transaction ends when claimBatch returns. Kafka publishing begins after the database locks have been released.
Recover expired claims
A separate short transaction returns abandoned work to PENDING:
update outbox_event
set status = 'PENDING',
claimed_by = null,
claimed_until = null,
next_attempt_at = now()
where status = 'PROCESSING'
and claimed_until < now();
An illustrative configuration for 120 order events per second might use:
outbox:
poll-interval: 500ms
batch-size: 200
claim-lease: 30s
publisher-instances: 3
max-attempts-before-alert: 10
retention: 7d
spring:
kafka:
producer:
acks: all
properties:
enable.idempotence: true
delivery.timeout.ms: 15000
request.timeout.ms: 5000
These are starting values, not universal recommendations. Size the batch, lease, and concurrency from measured acknowledgment latency, database capacity, per-instance throughput, and the delivery objective.
The lease must exceed normal batch processing time with margin. A fixed 30-second lease is unsafe if a valid batch can take 45 seconds. Long-running implementations should renew leases or use smaller batches.
Publish, acknowledge, then update
@Component
public class OutboxPublisher {
private final OutboxRepository outbox;
private final KafkaTemplate<String, String> kafkaTemplate;
private final MeterRegistry meters;
private final String instanceId = UUID.randomUUID().toString();
@Scheduled(fixedDelayString = "${outbox.poll-interval:500ms}")
public void publishBatch() {
Timer.Sample batchTimer = Timer.start(meters);
List<OutboxEventRow> events = outbox.claimBatch(instanceId, 200);
for (OutboxEventRow event : events) {
publishOne(event);
}
batchTimer.stop(meters.timer("outbox.publisher.batch.duration"));
}
private void publishOne(OutboxEventRow event) {
Timer.Sample sendTimer = Timer.start(meters);
try {
kafkaTemplate.send(
topicFor(event.eventType()),
event.aggregateId().toString(),
event.payload())
.get(10, TimeUnit.SECONDS);
outbox.markPublished(event.eventId(), instanceId, Instant.now());
meters.counter("outbox.publish", "result", "success").increment();
} catch (Exception failure) {
outbox.reschedule(
event.eventId(),
instanceId,
nextAttempt(event.attempts()),
rootMessage(failure)
);
meters.counter("outbox.publish", "result", "failure").increment();
} finally {
sendTimer.stop(meters.timer("outbox.kafka.ack.latency"));
}
}
}
Spring Kafka’s KafkaTemplate.send returns a CompletableFuture. This example blocks with a timeout to make the state transition easy to see.
An asynchronous implementation can improve throughput, but it must bound in-flight sends, match each completion to the correct event, and respect the claim lease.
The Kafka record key is aggregateId. Events for one order therefore map to one partition, provided the key and partition count stay stable. Kafka preserves order within a partition, not across an entire topic.
The publisher records success only after Kafka acknowledges the send. Failures return to PENDING with bounded exponential backoff and jitter:
Duration retryDelay(int attempts) {
long cappedSeconds = Math.min(60, 1L << Math.min(attempts - 1, 6));
double jitter = ThreadLocalRandom.current().nextDouble(0.8, 1.2);
return Duration.ofMillis((long) (cappedSeconds * 1000 * jitter));
}
The approximate base delays are 1, 2, 4, 8, 16, 32, and 60 seconds, with ±20% jitter.
Do not retry authorization, serialization, configuration, or incompatible-schema errors forever as if they were transient broker failures. Keep the row durable, classify the error, and alert an operator.
The duplicate window still exists
The outbox removes the missing-intent failure, but it cannot atomically combine these later actions:
- Kafka accepts the event.
- PostgreSQL marks the outbox row as
PUBLISHED.
Consider this sequence:
publisher -> Kafka: OrderCreated eventId=9c11... Kafka -> publisher: acknowledged publisher process: crashes outbox row: still PROCESSING lease expires next publisher -> Kafka: same eventId=9c11...
Kafka can receive the event twice. A timeout is also ambiguous: the client may not know whether the broker accepted the record.
The precise guarantee is:
The design provides recoverable, at-least-once event delivery. It does not provide exactly-once business processing by itself.
Kafka producer idempotence reduces duplicates caused by producer retries within a producer session. It does not remove the database status-update window across a process restart.
Make the consumer idempotent in its own transaction
The consumer can store stable event IDs:
create table processed_event (
consumer_name varchar(150) not null,
event_id uuid not null,
processed_at timestamptz not null,
primary key (consumer_name, event_id)
);
consumer_name allows independent business consumers to process the same event once each. Retention must cover the maximum plausible redelivery window. In this illustrative design, published outbox rows remain for 7 days and consumer deduplication records for 30 days.
The deduplication insert and business change belong in the same consumer-side database transaction:
@KafkaListener(topics = "order-created", groupId = "inventory")
@Transactional
public void onOrderCreated(EventEnvelope<OrderCreatedPayload> event) {
if (!processedEvents.tryInsert(
"inventory", event.eventId(), Instant.now())) {
return; // unique-key conflict: already applied
}
inventory.reserve(
event.payload().orderId(),
event.payload().totalAmount()
);
}
tryInsert can use INSERT ... ON CONFLICT DO NOTHING and report whether it inserted a row. If inventory reservation fails, the transaction rolls back the deduplication insert too. The Kafka offset must not be acknowledged as successfully processed until the database work commits.
Some transitions are naturally idempotent. Setting a projection to CREATED may be safe to repeat. Incrementing a balance or reserving stock usually is not.
Judge idempotency at the business operation—not only at the Kafka client.
A record key alone does not guarantee correct order
Using orderId as the Kafka key is necessary for per-order ordering, but it is not sufficient.
If OrderConfirmed sequence 2 is published while OrderCreated sequence 1 remains pending, Kafka will faithfully preserve the wrong publication order. The claim query therefore blocks later sequences while an earlier one remains unpublished.
Consumers can also store the last applied aggregate sequence and reject, defer, or investigate gaps. That protects against publisher defects and makes ordering failures observable.
Global ordering is normally unnecessary and expensive. The useful boundary here is one order.
Changing a topic’s partition count can also change the partition selected for a key. Teams with strict ordering requirements must account for that during topic changes and migrations.
Polling or CDC is an operational choice
The team started with polling because on-call engineers could inspect the table, run the claim query, and understand the recovery path at 2 a.m.
| Polling | CDC with Debezium |
|---|---|
| Easier to deploy and debug | Lower-latency streaming from the WAL |
| Poll interval adds latency | Requires Kafka Connect operations |
| Adds database queries and updates | Requires replication slots and WAL management |
| Recovery state is visible in the table | Recovery depends on connector offsets and lag |
| Application owns leases and cleanup | Connector owns capture; table retention still matters |
Debezium’s Outbox Event Router can capture outbox inserts and use the aggregate ID as the Kafka key. It also introduces connector configuration, replication-slot capacity, offset recovery, schema evolution, and lag monitoring.
CDC is not automatically the more mature design. For moderate traffic and a 10-second delivery objective, a 500-millisecond poll may be the more honest choice. At very high write rates or tight latency targets, CDC may justify its operational cost.
Monitor delay, not only row count
The most important alert is the age of the oldest ready or expired event:
select extract(epoch from (
now() - min(created_at)
)) as oldest_pending_seconds
from outbox_event
where status = 'PENDING'
or (status = 'PROCESSING' and claimed_until < now());
This metric answers the business question: how long has the rest of the system been waiting to hear about a committed change?
Useful metrics include:
outbox.oldest.pending.secondsoutbox.pending.count- Publish successes and failures
- Attempt count and backoff distribution
- Publisher batch duration
- Expired claim count
- Kafka acknowledgment latency
For the illustrative 10-second delivery objective, a warning might fire when the oldest event exceeds 30 seconds for five minutes, with a critical alert at 120 seconds. Real thresholds must come from the service-level objective and recovery capacity.
Total row count is a weak signal. A table can contain millions of healthy published rows with no delivery problem. It can also contain one old pending payment event that demands immediate attention.
Micrometer can expose the primary gauge:
Gauge.builder(
"outbox.oldest.pending.seconds",
outboxMetrics,
OutboxMetrics::oldestPendingAgeSeconds)
.description("Age of the oldest publishable or expired outbox event")
.register(meterRegistry);
Do not label this metric with orderId, eventId, or error text. Those high-cardinality values belong in structured logs and traces.
Cleanup belongs in the original design
An outbox table grows forever unless someone designs its end of life. Growth affects indexes, polling plans, vacuum behavior, backups, and storage.
This example marks records as PUBLISHED, retains them for seven days, and deletes them in small batches:
with victims as (
select event_id
from outbox_event
where status = 'PUBLISHED'
and published_at < now() - interval '7 days'
order by published_at
limit 5000
)
delete from outbox_event e
using victims v
where e.event_id = v.event_id;
Run the query repeatedly with pauses instead of deleting weeks of data in one transaction. Monitor duration, removed rows, dead tuples, table size, index size, and autovacuum behavior.
Never allow retention to delete PENDING or PROCESSING rows. Failed work is not old data.
Archiving may be necessary for audit or replay requirements. If replay is supported, make it an explicit, authorized operation. Retaining rows should not accidentally turn a cleanup incident into a replay.
Test the failure windows
Testcontainers can start real PostgreSQL and Kafka instances for integration tests:
@Testcontainers
@SpringBootTest
class OutboxIntegrationTest {
@Container
static PostgreSQLContainer<?> postgres =
new PostgreSQLContainer<>("postgres:17");
@Container
static KafkaContainer kafka =
new KafkaContainer(
DockerImageName.parse("apache/kafka:3.9.0")
);
@DynamicPropertySource
static void properties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", postgres::getJdbcUrl);
registry.add("spring.datasource.username", postgres::getUsername);
registry.add("spring.datasource.password", postgres::getPassword);
registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
}
}
One important failure-injection test simulates a crash after Kafka acknowledgment but before the status update:
@Test
void republishesSameEventIdWhenStatusUpdateFails() {
UUID orderId = createOrder();
UUID eventId = outbox.findByAggregateId(orderId).eventId();
outbox.failNextMarkPublished();
publisher.publishBatch(); // Kafka accepted; database update failed
clock.advance(Duration.ofSeconds(31));
publisher.recoverExpiredClaims();
publisher.publishBatch();
List<EventEnvelope<?>> received = kafkaRecords.forOrder(orderId);
assertThat(received).hasSize(2);
assertThat(received).extracting(EventEnvelope::eventId)
.containsOnly(eventId);
}
This test should expect a duplicate. If it expects exactly one Kafka record, it is testing a guarantee the design does not provide.
The failure suite should also cover:
- Kafka unavailable during order creation: the order and outbox row still commit.
- Kafka unavailable during polling: the event remains retryable.
- Publisher crash before acknowledgment: the lease expires and another publisher retries.
- Publisher crash after acknowledgment: the same stable event ID appears again.
- Two publishers claim concurrently: no active claim is shared.
- Duplicate consumer delivery: the business update occurs once.
- Multiple events for one order: sequence 2 never overtakes sequence 1.
- Cleanup during publication: no pending or claimed row is deleted.
Test recovery throughput too. A design that safely accumulates events during a one-hour Kafka outage but needs six hours to drain the backlog has not met a useful recovery objective.
What the team actually fixed
After the outbox was implemented:
- Creating an order committed the order and its outbox record together.
- Kafka downtime no longer erased publication intent.
- Pending events published after Kafka recovered.
- Publisher crashes could create duplicates, but consumers handled the stable event ID safely.
- Events remained ordered per order through the Kafka key and aggregate sequence.
- Operators detected delays through the oldest-pending-event age.
- Published rows were archived or deleted without removing pending work.
There was a cost. The service now owned another table, a publisher or CDC pipeline, a retry policy, consumer deduplication, monitoring, retention, and recovery procedures.
That was not accidental complexity. It was consistency work the original two-call method had left implicit.
Engineering lessons
A database commit is not distributed business success. It confirms only what that database committed.
@Transactionalcannot erase an independent commit boundary. PostgreSQL and Kafka still make separate decisions.Durable publication intent matters more than a long request retry. Once intent is stored, recovery no longer depends on one process or request lifetime.
At-least-once delivery requires business-level idempotency. A stable event ID helps only when the consumer protects its side effect transactionally.
Ordering, monitoring, cleanup, and recovery capacity belong in the design. They are not maintenance tasks to discover after events arrive out of sequence or the table grows without limit.
The practical result
The incident began with orders stuck in CREATED and a team convinced that a Kafka consumer was broken.
The consumers were fine. PostgreSQL was fine. Kafka recovered.
The architecture was the part that failed.
It allowed a committed business change to lose its integration event. The team fixed that gap by saving the order and publication intent inside one PostgreSQL transaction, then publishing asynchronously with leases, retries, stable event IDs, per-order sequencing, consumer deduplication, monitoring, and retention.
The result was not exactly-once processing, and it did not make PostgreSQL and Kafka atomic.
It replaced an unrecoverable missing event with visible, retryable work—and made the remaining duplicate window explicit.
That is the practical value of a transactional outbox.
