Postman Timed Out, but Kafka Was Still Receiving Jobs

Engineering Insights | By |
Postman Timed Out, but Kafka Was Still Receiving Jobs

A practical Spring Boot lesson on long-running batch requests, Kafka enqueue progress, and using SSE to reduce operational silence.

Quick Answer

A Postman timeout does not prove that Spring Boot stopped enqueueing Kafka jobs. Stream enqueue progress with Server-Sent Events, use stable operation and job IDs, and make retries idempotent so an operator can distinguish queued work from completed work.

Postman timed out.

Kafka was still receiving jobs.

Spring Boot was still working.

But the person running the request could no longer see what was happening.

That visibility gap was the real problem.

Server-Sent Events, or SSE, is a simple HTTP streaming mechanism that lets a server push text-based events to a client over one open connection.

In Spring Boot, a developer can use SSE to send progress updates while a request is still running.

This is useful when the backend operation is not instant, but the user still needs to know that the server is alive and moving.

In this scenario, Postman timed out while a Spring Boot endpoint was still enqueueing jobs into Kafka.

Kafka was receiving jobs.

Spring Boot was still working.

But the operator had no progress signal.

The Real Problem

The backend had one Spring Boot endpoint.

The endpoint accepted a list of jobs in one request.

For each job, the server sent a message to Kafka.

Kafka consumers processed those jobs asynchronously later.

The flow looked like this:

Postman
  -> Spring Boot endpoint
    -> loop through jobs
      -> send each job to Kafka
        -> Kafka consumers process jobs

For a small list, everything looked fine.

One job was fast.

Five jobs were fine.

But when someone sent 50, 100, or 300 jobs in one request, the operation took long enough that Postman, a gateway, or a server timeout could be reached.

Then Postman showed a timeout.

That raised difficult questions:

  • Did the request stop?
  • Were some jobs already queued?
  • Were all jobs queued?
  • Should the operator retry?
  • If they retry, will duplicate jobs be created?

That is where a simple timeout became risky.

Why Increasing Timeout Was Not Enough

The first instinct is usually to increase the timeout.

Increase Postman timeout.

Increase gateway timeout.

Increase server timeout.

Increase client read timeout.

Sometimes that helps for a while.

But it does not solve the real design issue.

Today, 50 jobs may finish inside the new timeout.

Tomorrow, 300 jobs may hit it again.

Next month, someone may send 1,000 jobs.

The better question is not:

“How do we make the request wait longer?”

The better question is:

“How do we show the operator that the server is still working?”

That is where SSE can help.

The Important Boundary

In this design, SSE is not used to wait until every Kafka job is fully processed.

That would make the HTTP request responsible for too much.

Kafka consumers may retry jobs.

Workers may fail.

Processing may take minutes or hours.

Some jobs may be delayed.

The HTTP request should not own that whole lifecycle.

The cleaner boundary is this:

HTTP endpoint receives the batch command.
Spring Boot enqueues each job into Kafka.
SSE reports enqueue progress to the operator.
Kafka consumers process the jobs asynchronously.

The SSE event means:

This job was accepted and queued to Kafka.

It does not mean:

This job has finished processing.

That distinction is important.

Example Setup

Assume the backend uses:

  • Java 21
  • Spring Boot
  • Spring Web
  • Spring Kafka
  • Apache Kafka
  • Postman for manual operation

Example Maven dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

spring-boot-starter-web gives us HTTP endpoints and SSE support.

spring-kafka gives us KafkaTemplate, which the service can use to send messages to Kafka.

Basic Kafka Configuration

A simple Kafka configuration may look like this:

spring.kafka.bootstrap-servers=localhost:9092

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

This config tells Spring Boot where Kafka is and how to serialize messages.

In real production systems, the developer should also configure retries, acknowledgements, delivery timeout, security, and observability.

For this article, the focus is the HTTP progress stream.

Request DTO

First, define the request body.

public record JobBatchRequest(
    String operationId,
    List<JobRequest> jobs
) {
}
public record JobRequest(
    String jobId,
    String type,
    Map<String, Object> payload
) {
}

The operationId identifies the whole batch request.

Each jobId identifies one job inside the batch.

The developer should use stable IDs because they help with logging, tracing, retry, and idempotency.

SSE Event DTO

Now define the event sent back to Postman.

public record JobEnqueueEvent(
    String operationId,
    String jobId,
    int sequence,
    int total,
    String status,
    String message,
    Instant timestamp
) {
}

This event gives the operator useful progress information.

For example:

{
  "operationId": "op-20260622-001",
  "jobId": "job-1001",
  "sequence": 1,
  "total": 100,
  "status": "QUEUED",
  "message": "Job queued to Kafka",
  "timestamp": "2026-06-22T10:00:00Z"
}

The operator can see which job was queued and how far the batch has progressed.

Kafka Message DTO

The Kafka message can be separate from the request DTO.

public record JobMessage(
    String operationId,
    String jobId,
    String type,
    Map<String, Object> payload,
    Instant queuedAt
) {
}

This message is what the consumer receives later.

It includes the operationId, so the developer can connect Kafka processing logs back to the original manual operation.

Spring Boot SSE Endpoint

Here is a simple controller using SseEmitter.

@RestController
@RequestMapping("/api/jobs")
public class JobBatchController {

    private final JobEnqueueService jobEnqueueService;

    public JobBatchController(JobEnqueueService jobEnqueueService) {
        this.jobEnqueueService = jobEnqueueService;
    }

    @PostMapping(value = "/enqueue", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public SseEmitter enqueueJobs(@RequestBody JobBatchRequest request) {
        SseEmitter emitter = new SseEmitter(0L);

        jobEnqueueService.enqueueWithProgress(request, emitter);

        return emitter;
    }
}

MediaType.TEXT_EVENT_STREAM_VALUE tells the client that this endpoint returns an SSE stream.

new SseEmitter(0L) disables the emitter timeout at the Spring MVC level.

That is useful for a demo, but production should be more deliberate.

Gateways, proxies, load balancers, and clients may still have their own timeouts. In production, use a timeout that matches the operation and infrastructure.

Service That Sends Jobs To Kafka

Now implement the enqueue service.

@Service
public class JobEnqueueService {

    private static final String TOPIC = "manual-job-requested";

    private final KafkaTemplate<String, JobMessage> kafkaTemplate;
    private final Executor taskExecutor;

    public JobEnqueueService(
            KafkaTemplate<String, JobMessage> kafkaTemplate,
            @Qualifier("applicationTaskExecutor") Executor taskExecutor) {
        this.kafkaTemplate = kafkaTemplate;
        this.taskExecutor = taskExecutor;
    }

    public void enqueueWithProgress(JobBatchRequest request, SseEmitter emitter) {
        taskExecutor.execute(() -> {
            List<JobRequest> jobs = request.jobs();
            int total = jobs.size();

            try {
                sendEvent(emitter, new JobEnqueueEvent(
                    request.operationId(),
                    null,
                    0,
                    total,
                    "STARTED",
                    "Batch enqueue started",
                    Instant.now()
                ));

                for (int i = 0; i < total; i++) {
                    JobRequest job = jobs.get(i);
                    int sequence = i + 1;

                    sendEvent(emitter, new JobEnqueueEvent(
                        request.operationId(),
                        job.jobId(),
                        sequence,
                        total,
                        "ENQUEUEING",
                        "Sending job to Kafka",
                        Instant.now()
                    ));

                    JobMessage message = new JobMessage(
                        request.operationId(),
                        job.jobId(),
                        job.type(),
                        job.payload(),
                        Instant.now()
                    );

                    kafkaTemplate.send(TOPIC, job.jobId(), message).get();

                    sendEvent(emitter, new JobEnqueueEvent(
                        request.operationId(),
                        job.jobId(),
                        sequence,
                        total,
                        "QUEUED",
                        "Job queued to Kafka",
                        Instant.now()
                    ));
                }

                sendEvent(emitter, new JobEnqueueEvent(
                    request.operationId(),
                    null,
                    total,
                    total,
                    "DONE",
                    "All jobs were queued to Kafka",
                    Instant.now()
                ));

                emitter.complete();
            } catch (Exception ex) {
                sendError(emitter, request.operationId(), ex);
            }
        });
    }

    private void sendEvent(SseEmitter emitter, JobEnqueueEvent event) throws IOException {
        emitter.send(SseEmitter.event()
            .name(event.status().toLowerCase())
            .data(event));
    }

    private void sendError(SseEmitter emitter, String operationId, Exception ex) {
        try {
            emitter.send(SseEmitter.event()
                .name("failed")
                .data(new JobEnqueueEvent(
                    operationId,
                    null,
                    0,
                    0,
                    "FAILED",
                    ex.getMessage(),
                    Instant.now()
                )));
        } catch (IOException ignored) {
            // Client may already be disconnected.
        } finally {
            emitter.completeWithError(ex);
        }
    }
}

This service sends progress events while it loops through the jobs.

After each successful Kafka send, it sends a QUEUED event back to the client.

The .get() waits for Kafka send confirmation. This makes the event more meaningful because QUEUED means Kafka acknowledged the send.

If the developer does not need that guarantee, they can use callbacks instead.

In production, be careful with blocking calls inside large loops. Use a bounded executor, sensible Kafka timeouts, and batch limits.

What Postman Sees

Postman can receive a streaming response.

The response may look like this:

event: started
data: {"operationId":"op-001","jobId":null,"sequence":0,"total":3,"status":"STARTED","message":"Batch enqueue started","timestamp":"2026-06-22T10:00:00Z"}

event: enqueueing
data: {"operationId":"op-001","jobId":"job-1","sequence":1,"total":3,"status":"ENQUEUEING","message":"Sending job to Kafka","timestamp":"2026-06-22T10:00:01Z"}

event: queued
data: {"operationId":"op-001","jobId":"job-1","sequence":1,"total":3,"status":"QUEUED","message":"Job queued to Kafka","timestamp":"2026-06-22T10:00:02Z"}

event: queued
data: {"operationId":"op-001","jobId":"job-2","sequence":2,"total":3,"status":"QUEUED","message":"Job queued to Kafka","timestamp":"2026-06-22T10:00:03Z"}

event: done
data: {"operationId":"op-001","jobId":null,"sequence":3,"total":3,"status":"DONE","message":"All jobs were queued to Kafka","timestamp":"2026-06-22T10:00:04Z"}

The important part is that the response is no longer silent.

The operator can see progress while the backend works through the list.

This reduces guessing.

Run Or Test With Curl

Postman can be used, but curl is also useful for testing streams.

curl -N \
  -X POST http://localhost:8080/api/jobs/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "operationId": "op-001",
    "jobs": [
      { "jobId": "job-1", "type": "REPORT", "payload": { "month": "2026-06" } },
      { "jobId": "job-2", "type": "REPORT", "payload": { "month": "2026-07" } },
      { "jobId": "job-3", "type": "REPORT", "payload": { "month": "2026-08" } }
    ]
  }'

The -N option disables buffering in curl.

That makes it easier to see events as they arrive.

If events only appear at the end, there may be buffering somewhere in the client, proxy, gateway, or server configuration.

Kafka Consumer Example

The Kafka consumer still processes jobs asynchronously.

@Component
public class JobConsumer {

    @KafkaListener(topics = "manual-job-requested", groupId = "job-worker")
    public void consume(JobMessage message) {
        System.out.println("Processing job: " + message.jobId()
            + ", operationId: " + message.operationId());

        // Run the real business logic here.
    }
}

This consumer is separate from the SSE endpoint.

The endpoint only reports enqueue progress.

The consumer owns job processing.

This keeps the system boundary clean.

Why This Helps Manual Operations

Before SSE, timeout made the operator think the process had stopped.

But the server might still be enqueueing jobs in the background.

That is dangerous because the operator may retry the same request.

If retry is not idempotent, duplicate jobs can be created.

If those jobs send reports, billing updates, notifications, corrections, or external API calls, the damage can spread.

After SSE, the operator can see progress.

For example:

Job 1 queued
Job 2 queued
Job 3 queued
Job 4 failed
Job 5 not started yet

This gives enough information to make a better decision.

The operation is still manual, but it is no longer silent.

Add Idempotency

SSE improves visibility, but it does not replace idempotency.

The developer should still make retries safe.

One simple approach is to store job requests by operation ID and job ID.

CREATE TABLE job_enqueue_log (
    operation_id VARCHAR(100) NOT NULL,
    job_id VARCHAR(100) NOT NULL,
    status VARCHAR(30) NOT NULL,
    queued_at TIMESTAMP,
    error_message TEXT,
    PRIMARY KEY (operation_id, job_id)
);

Before enqueueing a job, the service can check whether that (operation_id, job_id) was already queued.

If yes, it can skip or report it as already queued.

This prevents accidental duplicate messages when a user retries after a timeout.

Example Idempotency Check

A simplified flow can look like this:

if (enqueueLogRepository.existsQueued(request.operationId(), job.jobId())) {
    sendEvent(emitter, new JobEnqueueEvent(
        request.operationId(),
        job.jobId(),
        sequence,
        total,
        "SKIPPED",
        "Job was already queued",
        Instant.now()
    ));
    continue;
}

This code avoids sending the same job again if it was already queued.

In production, the developer should make the check and insert atomic, usually with a database constraint or transaction.

Idempotency is especially important when operators manually retry requests.

Watch Out For Proxy Buffering

SSE works best when events are flushed to the client quickly.

But some proxies buffer responses.

For example, Nginx may buffer upstream responses unless configured otherwise.

A typical Nginx setting is:

proxy_buffering off;

The exact configuration depends on the infrastructure.

The developer should test the full path:

Client -> gateway -> ingress -> service -> Kafka

Testing only against localhost may hide buffering or timeout behavior that happens in staging or production.

Important Production Notes

SSE keeps an HTTP connection open.

That has operational cost.

The developer should consider:

  • Maximum batch size
  • Request timeout
  • Gateway timeout
  • Connection limit
  • Thread pool size
  • Client disconnect behavior
  • Retry safety
  • Kafka send timeout
  • Back pressure
  • Authentication and authorization

Do not let one manual operation enqueue unlimited jobs.

Set a reasonable batch limit.

For example:

if (request.jobs().size() > 500) {
    throw new ResponseStatusException(
        HttpStatus.BAD_REQUEST,
        "Maximum batch size is 500 jobs"
    );
}

This protects the service from a request that is too large for the operational model.

Expected Result

After adding SSE progress events:

  • Postman no longer looks silent during long enqueue operations.
  • Operators can see which job is being processed.
  • Operators can see which jobs were queued to Kafka.
  • A timeout is less likely to be mistaken for total failure.
  • Retry decisions become safer.
  • Kafka consumers still process jobs asynchronously.
  • The HTTP boundary stays focused on enqueue progress, not full job completion.

The main benefit is not speed.

The main benefit is visibility.

Important Notes

Timeout does not always mean the work stopped.

A client timeout, gateway timeout, and server-side execution are different things.

The server may continue processing after the client gives up, depending on how the application is written.

For operations with side effects, this matters.

If users cannot see what happened, they may retry.

If retry is not safe, duplicates can happen.

SSE helps the system speak while it works, but it should be paired with idempotency, operation IDs, clear statuses, and useful logs.

Recommendation

For long-running batch enqueue operations, the developer should design three things:

  • Progress visibility
  • Retry safety
  • Clear async boundaries

A good response event should include:

operationId
jobId
sequence
total
status
timestamp
message

The event status should be precise.

Use QUEUED only when the job has been sent to Kafka.

Do not use COMPLETED unless the worker actually finished the job.

That wording matters because operators will make decisions based on it.

Related Production Lessons

FAQ

Does a Postman timeout stop a Spring Boot request?

Not necessarily. The client may stop waiting while server-side code continues running, depending on the application and infrastructure.

What should an SSE progress event confirm?

Use QUEUED only after Kafka acknowledges the send. Do not report COMPLETED until a consumer finishes the job.

How do you prevent duplicate Kafka jobs after a retry?

Use stable operation and job IDs with an atomic database constraint or transaction that records whether each job was already queued.

Conclusion

Postman timed out, but Kafka was still receiving jobs.

The system was not fully broken.

The operator just could not see what the system was doing.

That silence created uncertainty.

And uncertainty leads to risky retries.

In this solution, Spring Boot used SSE to stream progress events while the endpoint enqueued jobs into Kafka. The HTTP request reported enqueue progress, while Kafka consumers continued to own the real async processing.

SSE did not make Kafka faster.

It did not replace workers.

It did not turn an async process into a synchronous one.

It simply gave the operator a stream of progress events while Spring Boot worked through the batch.

Sometimes the best backend improvement is not a faster query, bigger pod, or longer timeout.

Sometimes it is teaching the system to speak while it works.

Related Posts