Overview
Kafka is a distributed commit log optimized for high-throughput, durable, ordered event streaming. Unlike traditional message queues, Kafka retains messages for a configurable period — consumers can replay, multiple consumer groups read the same topic independently, and partitions enable horizontal parallelism. Understanding the storage model (append-only log), replication (ISR), and the role of ZooKeeper/KRaft is foundational to every Kafka use case.
Architecture
KAFKA CLUSTER ARCHITECTURE
════════════════════════════════════════════════════════
┌────────────────────────────────────────────────────────┐
│ Kafka Cluster │
│ │
│ Broker 1 (Leader for P0, P2) │
│ ┌──────────────────────────────────────────────────┐ │
│ │ topic: orders partition: 0 offsets: 0...999 │ │
│ │ [msg0][msg1][msg2]...[msg999] → append-only │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ Broker 2 (Leader for P1, Follower for P0) │
│ ┌──────────────────────────────────────────────────┐ │
│ │ topic: orders partition: 1 (different data) │ │
│ │ replica of P0 (syncing from Broker 1) │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ Broker 3 (Follower for P0, P1) │
└────────────────────────────────────────────────────────┘
REPLICATION & ISR:
replication.factor = 3 → P0 on brokers 1, 2, 3
ISR (In-Sync Replicas) = replicas caught up to leader
Producer acks=all → leader waits for all ISR to confirm write
Broker failure: ISR elects new leader (only from ISR set)
min.insync.replicas=2 → refuse writes if fewer than 2 replicas in ISR
PARTITION ASSIGNMENT
════════════════════════════════════════════════════════
Without key: round-robin across partitions
With key: hash(key) % numPartitions → same key → same partition → ordered
Ordering guarantee: only within a partition
Total order: single partition (but limits throughput to 1 partition)
Per-entity order: use entity ID as key (all events for user-123 → partition 2)
LOG COMPACTION
════════════════════════════════════════════════════════
Normal retention: delete messages older than retention.ms (default 7 days)
Log compaction: keep only LATEST value per key (like a KV store)
Key=user-123: [update1][update2][update3] → compacted: [update3]
Key=user-456: [update1][delete-tombstone] → compacted: (deleted)
Use for: changelog topics (database table mirror), event sourcing replay
cleanup.policy=compact vs delete vs compact,delete
Technical Implementation
Topic Configuration for Durability vs Throughput
# Create topic with production durability settings
kafka-topics.sh --create \
--topic orders \
--partitions 12 \
--replication-factor 3 \
--config min.insync.replicas=2 \
--config retention.ms=604800000 \ # 7 days
--config max.message.bytes=1048576 \ # 1MB max message size
--config compression.type=lz4 \ # broker-side compression
--bootstrap-server kafka:9092
# Alter topic: increase partitions (can only increase, never decrease)
kafka-topics.sh --alter --topic orders --partitions 24 --bootstrap-server kafka:9092
# Check ISR health
kafka-topics.sh --describe --topic orders --bootstrap-server kafka:9092
# Output shows: Leader, Replicas, Isr — if Isr != Replicas, under-replicated!
Producer Durability Configuration
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092,kafka2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName());
// Durability vs throughput trade-offs:
props.put(ProducerConfig.ACKS_CONFIG, "all"); // wait for all ISR (safest)
// "1" = only leader (faster, can lose if leader dies before replicas sync)
// "0" = fire and forget (fastest, no durability guarantee)
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); // retry on transient failure
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // exactly-once producer
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // must be ≤ 5 for idempotence
// Throughput optimization:
props.put(ProducerConfig.LINGER_MS_CONFIG, 5); // wait 5ms to batch more messages
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 32768); // 32KB batch size
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // compress batches
KafkaProducer<String, OrderEvent> producer = new KafkaProducer<>(props);
// Send with callback (async)
producer.send(
new ProducerRecord<>("orders", order.getId(), new OrderEvent(order)),
(metadata, exception) -> {
if (exception != null) {
log.error("Failed to send order {}", order.getId(), exception);
} else {
log.debug("Sent to {}-{} offset {}", metadata.topic(), metadata.partition(), metadata.offset());
}
}
);
Consumer Group and Offset Management
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processor-v1"); // consumer group
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class.getName());
// Offset management:
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); // manual commit (safer)
// AUTO_COMMIT can commit before processing is complete → data loss on crash
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // or "latest"
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500); // max per poll()
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000); // 5min: processing SLA
KafkaConsumer<String, OrderEvent> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));
try {
while (running) {
ConsumerRecords<String, OrderEvent> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, OrderEvent> record : records) {
processOrder(record.value()); // your business logic
}
// Commit only after processing all records in the batch
consumer.commitSync(); // synchronous: blocks until broker confirms
// consumer.commitAsync(): faster, no retry on failure
}
} finally {
consumer.close(); // commits final offsets, triggers partition rebalance
}
Interview Preparation
Q: How does Kafka guarantee message ordering?
Kafka guarantees ordering only within a single partition. Messages with the same key are always sent to the same partition (via hash(key) % numPartitions), so all events for a given entity (user-id, order-id) arrive in the order they were produced. Across partitions, there's no global ordering. If you need total ordering across all messages, use a single partition — but this limits throughput to one consumer. Most real-world systems only need per-entity ordering, which is achieved by using the entity ID as the Kafka key.
Q: What is the ISR (In-Sync Replicas) and why does it matter?
The ISR is the set of replicas that are caught up to the leader's log within replica.lag.time.max.ms. When a producer sends with acks=all, the leader waits for all ISR members to acknowledge before responding. If a broker falls behind (slow disk, GC pause, network issue), it's removed from the ISR. The cluster continues operating, but with a smaller ISR — if ISR size drops below min.insync.replicas, the broker refuses writes (to prevent data loss). On broker recovery, the follower re-syncs with the leader and rejoins the ISR. ISR is the durability contract: data in ISR won't be lost even if the leader crashes.
Q: What is the difference between log compaction and time-based retention?
Time-based retention (retention.ms) keeps all messages up to a time limit, then deletes entire log segments. Good for event streams where you care about the history window (last 7 days of orders). Log compaction (cleanup.policy=compact) keeps only the latest message per key, regardless of age. A tombstone message (null value) signals deletion — compaction eventually removes both the tombstone and the key. Good for state change logs where you only need the current state per entity — like a table of user preferences or account balances. The compacted topic acts like a key-value store that can be replayed from the beginning to rebuild current state.
Learning Resources
- Apache Kafka Documentation
- Confluent — Kafka Internals
- Kafka: The Definitive Guide — Narkhede, Shapira, Palino
- Kafka Design — Replication — official design doc