TL;DR: Adding Kafka partitions seems like a quick win for throughput, yet it silently breaks Flink’s exactly-once guarantees. The fix is to pause the job, take a savepoint, reassign partitions, and restart with state migration plus an idempotent two-phase-commit sink.

Key Takeaways - Checkpoint barriers assume a stable partition-to-subtask map; changing that map breaks barrier alignment. - Offsets are stored per-partition in Flink’s distributed checkpoints; new partitions leave gaps that the two-phase commit cannot reconcile. - A safe scaling pattern - savepoint → reassign → restart with `partition.discovery` and an idempotent sink - restores exactly-once without downtime.

Adding Kafka partitions seems like a quick win for throughput, yet it can silently break Flink’s exactly-once guarantees. When you spin up extra partitions, the Flink job that once processed a tidy stream of records starts to see invisible hiccups. What hidden mechanism causes this failure? The surprise comes from the way Flink’s exactly-once contract is wired to the source parallelism. When the Kafka consumer is launched, each Flink subtask is assigned a fixed set of partitions. Flink’s checkpoint barrier travels along the same logical path as the data it protects. If you later reshuffle that path, the barrier can no longer guarantee that every record before the barrier has been checkpointed. How does this affect consistency? It breaks the assumptions baked into the source connector, the state backend, and the two-phase-commit sink. When those assumptions are violated, the job silently skips offsets or commits them twice. What deeper state interactions are at stake? The problem isn’t just about speed; it’s rooted in how Flink manages state across those partitions.

A Flink checkpoint consists of three moving parts.

1. Barrier propagation - each source subtask injects a barrier that travels downstream only after all upstream subtasks have emitted it.
2. Offset bookkeeping - the Kafka consumer stores the latest committed offset for each assigned partition inside the checkpoint snapshot.
3. Key-group distribution - keyed state is split into key-groups, each pinned to a specific subtask based on the job’s parallelism at the moment of checkpoint.

These pieces rely on a stable mapping between Kafka partitions and Flink subtasks. The source connector assumes that once a subtask owns partition P, it will always own P for the life of the checkpoint. The two-phase-commit sink (e.g., `TwoPhaseCommitSinkFunction`) expects that the transaction ID it opens corresponds to a deterministic set of offsets. What happens when that mapping changes? When you add partitions, the partition-to-subtask map changes. Existing subtasks keep their original assignments, while new subtasks appear to own the new partitions. The old checkpoints still contain offset entries only for the original partitions. The new partitions have no offset entry, so the sink cannot atomically commit a transaction that spans an unknown offset range. How does this lead to failures? This mismatch produces two concrete failure modes.

Barrier misalignment - some subtasks emit a barrier early, others lag because they now have extra partitions to poll. Downstream operators receive an incomplete barrier, causing the checkpoint to be marked failed or, worse, partially successful.

Offset gaps - the checkpoint snapshot lacks offsets for the new partitions, so on recovery the consumer may start from the earliest offset and re-process already-handled data. Which mode is more common in real jobs? Understanding these broken assumptions reveals the precise failure mode that jeopardizes exactly-once.

Imagine a job with three parallel subtasks, each consuming two partitions. You add six new partitions and let the connector auto-discover them. Subtask 0 now sees three partitions, while subtasks 1 and 2 still see two each. During the next checkpoint, subtask 0 must read from three partitions before it can emit the barrier, while the others finish after two. The barrier from subtask 0 arrives later, so downstream operators wait. The checkpoint coordinator eventually times out, marking the checkpoint as failed. What happens to offsets after this failure? When the checkpoint finally succeeds, the snapshot contains offsets for the old six partitions but none for the newly added three. The two-phase-commit sink opened a transaction at the start of the checkpoint, so it now has an incomplete view of the source state. On commit, the sink can only guarantee atomicity for the known offsets; the unknown ones remain pending. If the job crashes before the next successful checkpoint, Flink will restore from the last good snapshot, which lacks the new offsets, causing those records to be read again. The net effect is duplicate commits. How does this pattern impact the state backend? The state backend also suffers. Flink’s keyed state is partitioned into key-groups based on the job’s parallelism. Adding subtasks changes the key-group range each subtask owns. The old state stays in the original key-group slots, while the new subtasks start with empty state. When a checkpoint captures this mixed layout, the restored job may see state skew: some keys appear with stale values, others start from scratch. Can we restore consistency safely? These inconsistencies are subtle because the job continues to process records; the anomalies only surface as duplicate downstream writes or occasional data loss.

The recipe consists of five deterministic steps. Each step isolates the moving parts so that Flink’s assumptions stay intact.

1. Trigger a savepoint - freezes the entire job state.

```bash

flink savepoint :jobId /tmp/savepoints/savepoint-$(date +%s)

```

1. Pause the source - stops the Kafka consumer from pulling new records.

```java

kafkaConsumer.setStartFromGroupOffsets();

kafkaConsumer.pause();

```

1. Reassign partitions - use Kafka’s built-in tool to add or rebalance partitions.

```bash

kafka-reassign-partitions.sh \

--bootstrap-server broker1:9092,broker2:9092 \

--reassignment-json-file add-partitions.json \

--execute

```

1. Restart with the savepoint - enable discovery so Flink maps the new partitions to subtasks after the savepoint is restored.

```bash

flink run -d \

-p 6 \

--fromSavepoint file:///tmp/savepoints/savepoint-<your_savepoint> \

-c com.mycompany.StreamingJob \

my-job.jar \

-Dpartition.discovery.interval.millis=60000 \

-Dexecution.checkpointing.interval=30000

```

1. Use an idempotent two-phase-commit sink - ensures downstream writes survive retries without duplication.

```java

public class IdempotentKafkaSink extends TwoPhaseCommitSinkFunction<YourRecord, ProducerTransaction, ProducerRecord> {

@Override

protected void invoke(ProducerTransaction transaction, YourRecord value, Context context) {

transaction.send(new ProducerRecord<>(topic, value.getKey(), value));

}

@Override

protected void commit(ProducerTransaction transaction) {

transaction.commit();

}

}

```

With Kafka’s idempotent producer (`enable.idempotence=true`) and deterministic transaction IDs such as `jobId + checkpointId`, the sink can safely re-commit after a failure without duplicates.

Verification - after the job restarts, monitor the checkpoint log for “Barrier aligned” messages and query the Kafka consumer group offsets to confirm they now include the new partitions. Will this affect the job’s recovery time?

Zero data duplication eliminates costly downstream re-processing. When a sink writes each record exactly once, you can drop the “dedupe” stage that many teams build as a safety net. That reduction alone shrinks pipeline latency by several seconds per batch, which adds up to minutes over a day of streaming. In regulated domains such as healthcare, exactly-once semantics simplify compliance. HIPAA-compliant pipelines must guarantee that patient data is not replayed or lost. By guaranteeing atomic commits, you avoid the audit-heavy “re-ingest and reconcile” steps that usually inflate engineering effort. Speed translates to market-ready analytics. A pipeline that can safely add partitions without breaking guarantees lets you scale out as demand spikes, such as a new product launch or a sudden surge in IoT telemetry. The business can ship insights in weeks instead of months because the data team no longer needs to pause for manual offset reconciliation. Companies that adopt the safe scaling pattern report faster time-to-value and lower operational overhead. Will the extra steps introduce any hidden costs?

Q: Why does increasing Kafka partitions break Flink exactly-once?

A: More partitions disturb the alignment of checkpoint barriers and fragment offset bookkeeping, causing the two-phase commit protocol to lose its atomicity guarantees.

Q: Can I add partitions without stopping the Flink job?

A: You can, but you must use a savepoint, pause source consumption, reassign partitions, and restart with the saved state to keep exactly-once intact.

Q: What Flink configuration flags help mitigate partition-scaling issues?

A: Enable `partition.discovery.interval.millis`, set `execution.checkpointing.interval`, and use an idempotent `TwoPhaseCommitSinkFunction` to ensure consistent commits.

Q: Is exactly-once still possible with dynamic partition scaling?

A: Yes, by following a controlled rebalancing workflow: savepoint, Kafka re-partition, job restart, and idempotent sink handling.

Q: How does this pattern affect latency and throughput?

A: Latency spikes briefly during rebalancing, but overall throughput improves because the new partitions are fully utilized while exactly-once semantics remain intact.

Research and references cited in this article:

• How to Implement Flink Exactly-Once Processing
• Apache Flink does provide end-to-end exactly-once guarantees when coupled with d... | Hacker News
• Kafka | Apache Flink
• Exactly-once semantics in Flink Kafka Producer - Stack Overflow
• An Overview of End-to-End Exactly-Once Processing ... - Apache Flink
• State Management in Stream Processing: How Apache Flink and Kafka Streams Handle State
• Stateful Stream Processing | Apache Flink
• Flink State Management and Checkpointing | Conduktor
• Apache Flink: restoring state from checkpoint with changes Kafka topic
• State Rebuilds: Kafka Streams vs. Apache Flink - Acosom
• Kafka Streams vs Apache Flink: A Pragmatic Comparison ... - Medium
• Flink vs. Kafka and their role in the streaming data pipeline