A practical Spring Boot Kafka lesson on partition-level lag, slow consumer paths, and why total lag can hide the customer who is still waiting.
Kafka lag looked fine.
Reports were still late.
That was the confusing part.
The brokers were healthy. The consumer group was moving. Total lag did not look alarming.
But some customers were still waiting for reports.
The problem was hidden at the partition level.
Kafka consumer lag is the distance between the latest message available in a Kafka topic and the latest message processed by a consumer group.
In simple terms, lag tells the developer how far behind the consumer is.
But total lag is not always enough.
A Kafka topic is split into partitions. One partition can fall badly behind while the overall dashboard still looks acceptable from far away.
In this scenario, Kafka was not broken.
One partition was blocked by slow work inside the consumer path.
The Original Kafka Design
The reporting system used Kafka for report events.
The flow looked like this:
Report event -> Kafka topic: report-events -> Consumer group: report-service -> Spring Boot consumer -> Report generation logic
The system used customer_id as the Kafka message key.
That design made sense because report ordering mattered per customer.
For example, if customer C001 had multiple report events, the system wanted to process them in order.
Kafka uses the message key to choose a partition.
So events with the same customer_id go to the same partition.
customer_id = C001 -> partition 3 customer_id = C001 -> partition 3 customer_id = C001 -> partition 3
This preserves ordering for that key.
But it also creates a hidden trade-off.
If partition 3 becomes slow, every message behind the slow message in partition 3 has to wait.
The Symptom
Most reports were processed normally.
Some reports were delayed.
Kafka brokers looked healthy.
Throughput was still moving.
Total consumer lag looked acceptable.
There were no obvious red alerts.
So the first guesses were predictable:
Maybe Kafka needed more consumers.
Maybe the topic needed more partitions.
Maybe the brokers needed tuning.
Maybe reporting traffic was higher than usual.
Those were reasonable guesses.
But they were looking at the wrong level.
Why Total Lag Was Misleading
Total lag is a wide-angle metric.
It tells the developer whether the whole consumer group is falling behind.
But it can hide localized pain.
For example:
Partition 0 lag: 10 Partition 1 lag: 8 Partition 2 lag: 12 Partition 3 lag: 15,000 Partition 4 lag: 9
Most partitions look healthy.
One partition is in trouble.
If the system has many partitions and high traffic, total lag may not look scary enough at first glance.
But partition 3 has a real problem.
And customers mapped to partition 3 are real customers.
The dashboard was not lying.
It was just too aggregated.
What Partition-Level Lag Revealed
Once the team checked lag by partition, the picture changed.
Most partitions were moving normally.
One partition was drifting behind.
Customers mapped to that partition saw delayed reports.
The issue was not global.
It was localized, but still important.
The developer can check consumer group lag with:
kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --group report-service \ --describe
This command shows lag per topic and partition for the consumer group.
That detail matters because the slowest partition often tells a better story than the total lag number.
The Risky Consumer Path
The original Spring Kafka listener looked similar to this:
@KafkaListener(topics = "report-events", groupId = "report-service")
public void consume(ReportEvent event) {
validate(event);
// Risky: heavy query inside the ordered consumer path
ReportData data = reportRepository.loadLargeReportData(event.customerId());
reportService.generateReport(data);
}This code is simple, but it has a dangerous property.
The Kafka listener does heavy work directly inside the consumer path.
It validates the event.
Then it runs a large database query.
Then it generates the report.
If that database query is slow, the Kafka consumer is slow.
And if the consumer is slow for one message, the messages behind it in the same partition wait.
The Real Root Cause
The message did not fail.
It did not crash the consumer.
It did not cause endless retries.
It was just slow.
That is enough.
Kafka processes messages in a partition in order.
If one message takes a long time, the next message from that partition cannot safely jump ahead in the same consumer group.
So the root cause was not Kafka broker health.
It was not total system throughput.
It was a slow database query inside the ordered consumer path.
Kafka was preserving the ordering guarantee the system asked for.
Why Adding More Consumers Did Not Fix It
Adding more consumers can help when there are unassigned partitions or when work is spread across many partitions.
But in one consumer group, a partition is assigned to only one consumer at a time.
Another consumer cannot simply take the next message from the same partition while the assigned consumer is stuck.
For example:
Topic: report-events Partitions: 5 Consumer group: report-service Consumers: 5 Partition 3 -> Consumer B
If Consumer B is stuck processing a slow message from partition 3, another consumer cannot process the next message from partition 3.
That is how Kafka preserves order inside the partition.
More consumers may sit idle while the blocked partition remains blocked.
The Dangerous Pattern
The dangerous pattern is putting unpredictable slow work inside a Kafka listener.
For example:
Heavy database queries Long report generation Large file exports Slow external API calls Complex business workflows Large object transformation
This kind of work can turn one Kafka message into backpressure for the whole partition.
The developer should be extra careful when ordering matters.
Ordering gives correctness, but it also means messages share fate.
Better Design: Keep The Consumer Path Small
A safer Kafka consumer path should be small and predictable.
The consumer should:
- Receive the Kafka event.
- Validate basic fields.
- Persist minimal state if needed.
- Create a separate reporting job.
- Commit the Kafka offset after the handoff is safe.
The consumer should avoid doing heavy reporting work directly.
The listener can be changed to this:
@KafkaListener(topics = "report-events", groupId = "report-service")
public void consume(ReportEvent event, Acknowledgment ack) {
validate(event);
reportJobService.createJobIfAbsent(
event.eventId(),
event.customerId(),
event.reportMonth()
);
ack.acknowledge();
}This version does not generate the report inside the Kafka listener.
It creates a report job and acknowledges the Kafka message after the handoff succeeds.
The heavy work moves somewhere else.
Add Manual Ack Configuration
If the developer wants to manually acknowledge offsets, configure Spring Kafka like this:
spring.kafka.consumer.enable-auto-commit=false spring.kafka.listener.ack-mode=manual
This gives the application control over when the offset is committed.
The important rule is simple:
Do not acknowledge before the handoff is safe.
If the consumer acknowledges too early and job creation fails, the message may be lost.
If the consumer acknowledges too late, the message may be reprocessed.
That is why idempotency matters.
Idempotent Job Table
A job table can make the handoff safe and retryable.
CREATE TABLE report_job (
id BIGSERIAL PRIMARY KEY,
event_id VARCHAR(100) NOT NULL UNIQUE,
customer_id VARCHAR(100) NOT NULL,
report_month DATE NOT NULL,
status VARCHAR(30) NOT NULL,
retry_count INT NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);The event_id is unique.
If the Kafka message is processed twice, the service can avoid creating duplicate jobs.
That makes retry safer.
The developer can treat Kafka message handling as:
Create the job if it does not already exist.
Worker Processing
The heavy reporting logic can run in a separate worker.
For example:
@Scheduled(fixedDelay = 5000)
public void processReportJobs() {
List<ReportJob> jobs = reportJobRepository.findPendingJobs();
for (ReportJob job : jobs) {
reportWorker.process(job);
}
}This worker picks pending jobs and processes them outside the Kafka listener.
Now a slow report job can still be slow, but it does not block Kafka partition consumption for a long time.
The worker can have its own retry rules, dead-letter handling, concurrency model, and monitoring.
Example Worker Logic
A worker may look like this:
public void process(ReportJob job) {
try {
reportJobRepository.markRunning(job.id());
ReportData data = reportRepository.loadLargeReportData(job.customerId());
reportService.generateReport(data);
reportJobRepository.markCompleted(job.id());
} catch (Exception ex) {
reportJobRepository.markFailedOrRetry(job.id(), ex.getMessage());
}
}The slow database query still exists.
But it is no longer inside the ordered Kafka consumer path.
That is the key design change.
The developer should still optimize the query, but the blast radius is smaller.
Offset Commit Boundary
The offset commit boundary is important.
A safe pattern is:
1. Receive Kafka event. 2. Validate event. 3. Create report job idempotently. 4. Commit Kafka offset. 5. Worker processes report job separately.
This means Kafka ingestion is responsible for accepting work.
The worker system is responsible for completing work.
Those are two different responsibilities.
This separation makes monitoring and scaling easier.
Watch For Hot Keys
Using customer_id as the key may be correct when ordering matters per customer.
But it can also create hot partitions.
If one customer produces much more traffic than others, that customer’s partition can become heavier than the rest.
The developer should check:
- Which keys map to the slow partition
- Whether one customer or report type dominates the lag
- Whether ordering is truly required for every event
- Whether the key should include another dimension
- Whether the workflow needs a separate topic or processing path
Do not change the Kafka key casually.
Changing the key can change ordering guarantees.
But the key choice should be treated as a real architecture decision, not a default setting.
Monitoring After The Fix
After moving heavy work out of the consumer path, monitoring should be split into two parts.
Track Kafka ingestion:
kafka_consumergroup_lag{topic, group, partition}
consumer_processing_duration_seconds
consumer_errors_total
consumer_rebalance_total
offset_commit_failure_totalTrack worker processing:
report_worker_backlog_total report_worker_job_age_seconds report_worker_failure_total report_worker_retry_total report_worker_processing_duration_seconds database_query_duration_seconds
This matters because the bottleneck may move.
Kafka lag may improve, but worker backlog may grow.
That is not a failure of the design.
It is better visibility.
Now the system can show where the delay really lives.
Practical Debugging Checklist
When Kafka dashboards look fine but customers still see delays, ask:
- Is total lag hiding one bad partition?
- Which partition has the highest lag?
- Which keys map to that partition?
- Which customer or report type is affected?
- What is the consumer processing time per message?
- Is the consumer waiting on the database?
- Are heavy queries running inside the Kafka listener?
- Are more consumers actually useful for this partition count?
- Is ordering required for this exact work?
- Can heavy work be moved to a worker queue?
- Is the handoff idempotent?
- Are worker backlog and job age monitored?
This checklist helps move the investigation from “Kafka is slow” to “which part of the pipeline is slow?”
That is usually a better question.
Expected Result
After moving heavy work out of the Kafka consumer path:
- Kafka partition lag becomes easier to control.
- One slow report job no longer blocks the partition for a long time.
- Consumers spend less time inside database-heavy logic.
- Worker backlog becomes visible and separately scalable.
- Delayed customers are easier to identify.
- Monitoring separates Kafka ingestion from report processing.
The system does not magically make heavy reports fast.
But it prevents slow report generation from holding Kafka partitions hostage.
Important Notes
Total Kafka lag can hide partition-level problems.
Adding more consumers does not fix one blocked partition.
A partition can only be consumed by one consumer in the same group at a time.
customer_id as a key preserves ordering, but it also creates shared fate for customers in the same partition.
Do not put unpredictable slow work inside a Kafka listener.
Moving work to workers requires idempotency and retry design.
Monitor both Kafka lag and worker backlog, or the bottleneck may simply move.
Recommendation
Keep the Kafka listener fast, predictable, and easy to observe.
A good listener should usually do only enough work to safely accept the event and hand it off.
Heavy work should run in a worker model that can be retried, scaled, and monitored separately.
Also, do not rely only on total lag dashboards.
For Kafka systems, always keep partition-level lag visible.
The customer who is waiting is often hidden in the partition that the aggregate metric smoothed away.
Conclusion
Kafka was not broken.
The brokers were healthy.
The consumer group was still moving.
But one partition was falling behind because a slow database query sat inside the consumer path.
Total lag hid the pain.
The fix was not simply “add more consumers.”
The better fix was to check partition-level lag, identify the slow consumer path, and move heavy report generation into a separate worker flow with an idempotent handoff.
Kafka did exactly what it was supposed to do.
It preserved order.
The lesson is that ordering has a cost, and slow work inside that ordered path can delay everyone behind it.
Sometimes the production issue is not the whole system failing.
Sometimes it is one partition, one slow message, one heavy query, and one dashboard that averages away the customer who is still waiting.
