Kafka Interview Questions

Kafka is a distributed append-only log. Producers append records to a topic; the broker keeps every record on disk for a configured retention period (default 7 days). Multiple consumer groups can read the same topic independently, each tracking its own offset. Nothing is "consumed" or deleted on read.

RabbitMQ / SQS are queues: a message is routed to a consumer, acked, and removed. Once it's gone, no other consumer sees it.

• Topic — a named stream of records. Logical grouping (e.g., orders, payments).
• Partition — an ordered, immutable sequence of records inside a topic. A topic has N partitions; each partition lives on a broker. Partitioning is what makes Kafka horizontally scalable.
• Offset — a monotonically increasing 64-bit ID assigned to each record within a partition. Once written, the offset never changes.

Records are ordered within a partition, never across partitions. So if order matters (a user's events, a single order's lifecycle), they must all land on the same partition — usually by using the same key.

Topic: orders   (4 partitions, replication factor 3)

P0:  [m0][m1][m2][m3]...           offsets 0..3
P1:  [m0][m1][m2]...               offsets 0..2
P2:  [m0][m1][m2][m3][m4][m5]...   offsets 0..5
P3:  [m0][m1]...                   offsets 0..1

producer.send(new ProducerRecord("orders", "order-42", payload))
   // hash("order-42") % 4  →  partition 2
   // → broker that owns P2 → appended at next offset

1. Serialize the key and value (StringSerializer, AvroSerializer, etc.).
2. Partition — if you set a key, partition = murmur2(key) % numPartitions. No key → sticky partitioner picks a partition and sticks with it for the current batch.
3. Append to in-memory record accumulator — one queue per partition. The send() call returns a Future immediately.
4. I/O thread (Sender) drains accumulator batches when either: batch.size bytes filled, or linger.ms elapsed since the first record arrived.
5. Send to leader broker. The broker writes to the partition log, replicates per acks, returns ack.
6. If the request fails (timeout, NotLeaderForPartition), the producer retries per retries + retry.backoff.ms.

linger.ms=5            # wait up to 5ms to batch — single biggest throughput knob
batch.size=65536      # max batch bytes per partition (64KB is a sane default)
compression.type=lz4  # compress the batch — 3-5x throughput gain over none
acks=all              # durability — see section 4
enable.idempotence=true   # de-dup retries — see section 10
max.in.flight.requests.per.connection=5

For a payment service: acks=all + min.insync.replicas=2 + enable.idempotence=true. You'll pay 1–2ms more latency, but you'll never lose a transaction record on a single broker failure.

For metrics / clickstream where one in a million dropped events is fine, acks=1 is reasonable. acks=0 is for "we'd rather drop the data than slow the producer" — IoT telemetry at the edge, that kind of thing.

• A consumer group is a set of consumers that share a group.id.
• For each topic the group subscribes to, every partition is assigned to exactly one consumer in the group.
• If numConsumers < numPartitions: some consumers get multiple partitions.
• If numConsumers == numPartitions: 1 partition each — maximum parallelism.
• If numConsumers > numPartitions: extras sit idle. Partitions are the upper bound on consumer parallelism.

Auto-commit (enable.auto.commit=true): every auto.commit.interval.ms (default 5s), the consumer commits the latest polled offset — regardless of whether you finished processing those records. If you crash mid-batch, the offset has already moved past records you didn't actually process. You get at-most-once with potential data loss, or duplicates on the previous batch.

Manual commit with commitSync() or commitAsync() after processing: you commit only after the work is done. If you crash, you re-read and re-process. You get at-least-once with possible duplicates — the standard pattern.

props.put("enable.auto.commit", "false");
try (KafkaConsumer<String, String> c = new KafkaConsumer<>(props)) {
  c.subscribe(List.of("orders"));
  while (running) {
    ConsumerRecords<String, String> recs = c.poll(Duration.ofMillis(200));
    for (var r : recs) processOrder(r);  // idempotent!
    c.commitSync();                    // commit AFTER processing
  }
}

Triggers: a consumer joins (deploy), leaves (shutdown), times out (session.timeout.ms with no heartbeat), or topic metadata changes (partition count grew).

Eager (default before 2.4): all consumers stop polling, give up all partitions, group coordinator reassigns, all consumers resume. Total stop-the-world. A 200-consumer group can pause for seconds.

Cooperative-sticky (default 3.0+): only the partitions that need to move are revoked; everyone else keeps polling. Massively reduces pause duration. Set partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor.

Replication factor = how many copies of each partition exist across brokers. RF=3 means 3 copies on 3 different brokers.

Leader / Followers — for each partition, exactly one replica is leader. All reads and writes go to the leader. Followers pull from the leader to stay in sync.

ISR (In-Sync Replicas) — the subset of replicas that are caught up with the leader within replica.lag.time.max.ms (default 30s). A follower that falls behind is removed from ISR. The leader itself is always in ISR.

Leader failure: controller picks a new leader from the current ISR. If unclean.leader.election.enable=false (default since 2.5), and ISR is empty, the partition goes unavailable rather than electing a non-ISR replica that may have stale data.

Not quite. acks=all means "wait for every replica currently in ISR". If two of your three replicas have fallen behind and dropped out of ISR, then "all ISR" = just the leader. You acked from one box. If that box dies before any follower catches up, you lose data.

min.insync.replicas=2 on the topic forces the producer to fail with NotEnoughReplicasException rather than silently downgrading durability. The producer can retry; meanwhile no record is acked unless at least 2 replicas have it.

# Topic config
replication.factor=3
min.insync.replicas=2

# Producer config
acks=all
enable.idempotence=true   # de-dup retries (section 10)

Without idempotence, retries can produce duplicates: the broker received the original send, replied with an ack that got lost, the producer retried, the broker wrote the message a second time.

Idempotent producer adds three pieces:

• Producer ID (PID) — assigned by the broker on connect. Stable for the producer's lifetime.
• Sequence number per (PID, partition) — monotonically increasing.
• Broker-side de-dup window — broker tracks the highest sequence per PID per partition; rejects out-of-order or duplicate sequences.

Result: even if the producer retries the same batch 5 times, the broker writes it once.

Yes — for the read-process-write loop within Kafka. End-to-end EOS to a non-Kafka sink (a database, an HTTP API) requires your application to make the sink-side write idempotent or transactional.

The transactional producer adds:

• transactional.id — a stable identifier across producer restarts. Kafka uses it to fence zombie producers.
• initTransactions(), beginTransaction(), send(), sendOffsetsToTransaction(), commitTransaction() — APIs that make a batch atomic across multiple topic-partitions.
• Read-process-write atomicity: in one transaction, you read input from topic A, write output to topic B, and commit your input offsets — all atomically. Either everything happens or nothing does.

Consumers must set isolation.level=read_committed to skip records belonging to aborted transactions.

producer.initTransactions();
while (running) {
  ConsumerRecords<String,String> recs = consumer.poll(...);
  producer.beginTransaction();
  for (var r : recs) {
    producer.send(new ProducerRecord("output", transform(r.value())));
  }
  producer.sendOffsetsToTransaction(offsetsOf(recs), consumer.groupMetadata());
  producer.commitTransaction();   // atomic: writes + offset commit
}

Hot partition = one partition receiving disproportionate traffic because the partition key has a skewed distribution. Symptoms: one consumer falls behind, one broker is hot, others are cool.

• Pick a high-cardinality key. order_id over country. user_id over region.
• Salt the key if cardinality is bounded. Append a random 0–N suffix to artificially split a hot key — at the cost of losing per-key ordering for that hot key.
• Custom partitioner. If your domain has known hot keys, route them to a sub-partition pool.
• Match key to ordering needs. If order matters per order_id, use order_id; don't pick country for "scale" and then complain about ordering.

Kafka guarantees ordering within a single partition. Records appended to partition P0 are read in append order. Period.

Across partitions there is no global order. Two records in P0 and P1 may be read in either order by a consumer reading both.

So: if you need "events for order-42 in order", make sure they all go to the same partition. The producer does this automatically when you set key=order-42 — every record with that key is hashed to the same partition.

Without idempotence: yes. With 5 in-flight batches and retries enabled, batch 1 can fail and be retried, while batches 2–5 already succeeded. Now batch 1's record arrives after batch 2's — out of order.

With enable.idempotence=true: no. The broker uses sequence numbers per (PID, partition) and rejects out-of-order writes; the producer waits and reorders. As of Kafka 3.0, idempotence allows up to 5 in-flight requests safely (older versions required ≤ 5 only without idempotence; 1 with).

Lag = log_end_offset − consumer_offset, per partition. It's how many records the consumer is behind the producer's latest write.

Common spike causes:

• Producer burst — Black Friday, viral spike, retried fan-out from upstream.
• Slow downstream — DB call inside the consumer suddenly takes 200ms instead of 5ms.
• Rebalance — paused all consumers for a few seconds; backlog built up.
• Hot partition — one partition's lag spikes while others are fine; key skew (section 13).
• Consumer crash / GC pause — JVM full GC stalled poll() for 30s.
• Network throttling / quota hit — the broker is shaping your consumer.

Without schemas, every consumer parses raw bytes and prays the producer didn't change anything. Add a field, drop a field, rename — every downstream consumer breaks silently. Schema Registry stores versioned schemas keyed by topic-subject; producers serialize with the registered schema ID embedded in the message; consumers fetch the schema by ID to deserialize.

Compatibility modes (set per subject):

• BACKWARD (default) — new schema can read data written by the previous schema. Producers can upgrade; consumers stay on old schema.
• FORWARD — old schema can read data written by the new schema. Consumers can upgrade; producers stay on old.
• FULL — both directions. Adding optional fields with defaults is the safe lane.
• NONE — anything goes. Don't.

Use a plain consumer when you're doing stateless work: read a record, transform it, write to a DB or call an API. No joins, no aggregations across records.

Use Kafka Streams when:

• You need to join two streams (orders ⨝ payments).
• You need windowed aggregations (5-minute rolling sum of clicks).
• You need state and want it backed by Kafka (changelog topic + RocksDB local store).
• You want exactly-once processing across multi-step pipelines for free.

Streams primitives: KStream (event stream), KTable (changelog / current value per key), GlobalKTable (broadcast lookup table). State stores back KTables and aggregations; they're persisted as compacted topics so a restarted node can rebuild state.

Kafka Connect runs as a separate cluster of workers that host plug-in connectors. Source connectors pull from MySQL/Postgres/MongoDB/S3 into Kafka; sink connectors push from Kafka to Snowflake/Elasticsearch/JDBC. You configure them with JSON over a REST API; no code.

A poison pill is a record the connector cannot process — bad schema, parse error, FK violation. Without DLQ config, the connector retries forever and stalls the entire partition.

Configure errors.tolerance=all + errors.deadletterqueue.topic.name=my-connector-dlq and bad records get routed to a separate topic for manual triage; the main pipeline keeps moving.

ZooKeeper held controller election state and topic metadata. Operating two distributed systems (Kafka + ZK) was painful: separate config, separate JVM tuning, separate failure modes, slow metadata propagation.

KRaft bakes the same metadata-replication and controller-election protocol into Kafka itself, using the Raft consensus algorithm. A small set of broker nodes act as controllers (analogous to ZK ensemble); they replicate the metadata log among themselves; clients no longer need any external coordinator.

Benefits: faster controller failover, faster cluster start-up, simpler operations, supports far more partitions per cluster (millions vs ~200K).

Sequential writes (no random seeks), zero-copy via sendfile() from page cache straight to socket buffer (no userspace copy), batching at every layer, and reliance on OS page cache rather than a custom buffer pool. Disk is only "slow" for random I/O — Kafka does pure append.

Yes. consumer.seek(partition, offset) after assignment. Or seekToBeginning() / seekToEnd(). This is how you implement "replay last hour" without changing groups.

With delete.topic.enable=true (default), the topic is marked for deletion and removed asynchronously by the controller. In-flight producer sends fail with UnknownTopicOrPartitionException. Always drain producers first.

After joining the group, the group coordinator broadcasts the assignment via SyncGroupResponse. The consumer then calls onPartitionsAssigned and starts polling those partitions. On rebalance, it gets onPartitionsRevoked first, then a new assignment.

Yes — the leader broker serializes them. Each producer's request is appended in arrival order. Within a single producer, ordering is preserved by sequence numbers (idempotence). Across producers, "first write wins" is at-the-broker, not coordinated.

Empty poll = no new records on assigned partitions in poll(timeout). No partitions = the group has more consumers than partitions; coordinator gave you nothing. Check consumer.assignment() to distinguish.

6 instances each get 1 partition. The other 6 are standby replicas for state stores (if num.standby.replicas>0) or fully idle. They take over on failure but contribute no processing capacity. Same partition cap as plain consumers.

Transactions require idempotence — setting transactional.id implicitly enables enable.idempotence=true. The two work together; the question is whether you also need transactional atomicity beyond just retry-safety.

A zombie is a producer instance you thought was dead but is actually still running (e.g., GC'd long enough to be considered dead, then woke up). With transactional.id, the broker tracks an epoch per ID — when a new instance starts with the same transactional.id, it bumps the epoch and any send from the old (lower-epoch) producer is rejected with ProducerFenced. That's how exactly-once survives instance failures.

No within a partition — offsets are monotonically increasing and immutable once written. But a consumer's committed offset for a group can be moved backward via seek + commitSync, which is how "replay" works.