Kafka Architecture Explained

Key Takeaways

• Kafka stores events in partitioned, replicated topic logs. Ordering is defined within a partition, not across a topic.
• Producers publish independently of consumers. Consumers pull records, and reading a record doesn't delete it.
• Within a standard consumer group, each partition has at most one assigned consumer; one consumer can read several partitions.
• Replication protects stored records, while committed consumer offsets provide restart checkpoints. Neither automatically makes external processing exactly-once.
• Kafka 4.x uses KRaft controllers for cluster metadata. ZooKeeper mode was removed in Kafka 4.0.

Kafka architecture centers on a durable, distributed log: producers append records to topic partitions, brokers store and replicate those partitions, and consumer groups read them independently. Partitions provide the boundaries for ordering and parallelism; retention separates the lifetime of records from the pace of individual consumers.

The Apache Kafka introduction describes this retained-log model. This article explains its moving parts using standard consumer groups; for the architectural decision itself, see when to use Kafka.

Kafka Architecture Explained

Brokers store data; KRaft controllers manage metadata

A broker is a Kafka server that stores partition replicas and handles client requests. Controllers manage cluster metadata, including broker registrations, partition assignments, and partition leadership. In KRaft, controllers replicate that metadata through a Raft-based metadata log.

The KRaft controller quorum has an active controller and standby controllers. A majority is required for the quorum to make progress. Broker and controller roles can share a process, but separate roles provide isolation for critical deployments.

                   KRaft metadata quorum
               [C1 active] [C2 standby] [C3 standby]
                          |
                 metadata / leader elections
                          |
Topic: orders       Broker B1   Broker B2   Broker B3
Partition P0        leader      follower    follower
Partition P1        follower    leader      follower
Partition P2        follower    follower    leader

Example: replication factor = 3; all replicas healthy
ISR for each partition = {B1, B2, B3}

Producers --> partition leaders
Consumers --> brokers, using fetch requests

Controllers aren't an intermediary for application records: clients communicate with brokers. Kafka 4.0 removed ZooKeeper mode, so older ZooKeeper-based clusters must migrate to KRaft before upgrading to Kafka 4.x; the upgrade guide documents that requirement.

Kafka Topics, Partitions, and Commit Logs

A topic is a named stream split into partitions

A topic such as orders groups related events. Its partitions are separate ordered logs distributed across brokers, with replicas providing additional copies. Topic-level configuration controls policies such as retention and can override broker defaults.

A partition's log is stored in segment files. New records append to the active segment; Kafka rolls new segments and later cleans up older ones. Records aren't edited in place, but cleanup can remove them. The log implementation documentation explains segment storage.

Each record has a numerical offset within its partition. An offset isn't a topic-wide identifier or a count of retained records. Compaction can leave gaps without renumbering the remaining records. Also, a partition's “commit log” and a consumer's “offset commit” are different concepts: one stores events; the other stores a checkpoint.

Partitions define ordering, not business time

Kafka preserves the order of records as written within a partition, not across all partitions of a topic. To keep an order's events together, you might use its order ID as the record key. That requires consistent partitioning; Kafka doesn't independently sort events by their timestamps.

Bytes need a shared data contract

Producer serializers turn keys and values into bytes; consumer deserializers turn them back into application data. JSON, Avro, and Protocol Buffers are payload-format choices, not separate broker storage architectures. Kafka's serialization documentation explains this boundary.

A schema defines the fields and types both sides expect. Because producers and consumers deploy independently, test compatibility before changing that contract—including compatibility with retained events you may replay. Formats have different evolution rules; Avro's schema-resolution specification is one concrete example.

Kafka Brokers and Replication

Each partition has a leader replica and follower replicas. Producers send writes to the leader; followers fetch and replicate its log. The in-sync replica set, or ISR, identifies replicas sufficiently caught up under Kafka's replication rules.

If a leader fails, availability depends on electing a suitable replacement—not merely having another copy somewhere. An unavailable or stale replica may not be able to take over safely. Allowing unclean leader election can restore availability at the cost of losing records.

Current Kafka also supports eligible leader replicas: replicas outside the current ISR that Kafka knows are safe leadership candidates. Election behavior therefore depends on the cluster's enabled features and policy, not just the replication factor.

Kafka's append-oriented storage and OS page cache help avoid unnecessary random I/O and repeated disk reads. The page cache is memory caching, not an alternative to using memory. Actual throughput and latency depend on batching, storage, network capacity, replication, and workload; the Kafka design documentation explains these mechanisms.

Producers Explained: Partitioning, Batching, and Delivery

Partition selection isn't universally round-robin

The current Java producer's default behavior distinguishes these cases:

• A key is present: hash the key to select a partition, unless an explicit partition or different partitioning configuration overrides that choice.
• No key or explicit partition: use sticky partitioning, keeping records together to build batches before switching partitions.
• Custom routing: supply an explicit partition or a custom partitioner.

Producers batch records by partition. batch.size controls batch sizing, while linger.ms permits a batching delay. Larger batches can reduce request overhead, but waiting to fill them trades latency for efficiency. These behaviors and controls are documented in the producer configuration reference.

Acknowledgments and retries have different jobs

For example, configure a topic with three replicas and min.insync.replicas=2, and produce with acks=all. Writes can succeed with two in-sync replicas but fail with only one. If all three replicas are in sync, acks=all waits for all three—not just the configured minimum of two. See the broker configuration reference.

# Java producer configuration fragment, not a complete client setup
acks=all
enable.idempotence=true

# Separate topic configuration; create the topic with three replicas
min.insync.replicas=2

Retries recover from transient send failures. Idempotent production prevents the producer's protocol-level retries from appending duplicate copies. It doesn't deduplicate arbitrary application-level resends or external side effects. The current Java producer enables idempotence by default when configuration doesn't conflict; the KafkaProducer documentation explains its session scope and limitations.

Kafka Consumer Groups and Pull-Based Consumption

Groups divide partitions, not individual records

Consumers sharing a group.id form a consumer group. With automatic group assignment, a partition is assigned to at most one member at a time, while a member can own multiple partitions. For one subscribed topic with three partitions, more than three consumers cannot increase assignment-level parallelism.

Topic: orders             Example assignments

P0 --------------------> Group A / Consumer A1
P1 --------------------> Group A / Consumer A1
P2 --------------------> Group A / Consumer A2

P0, P1, P2 ------------> Group B / Consumer B1

Group A and Group B maintain independent progress.

Separate groups can each read the same retained records independently. One group therefore provides queue-like work sharing; multiple groups provide independent subscriptions. The KafkaConsumer documentation describes these assignment rules. For a broader messaging comparison, see JMS vs Kafka.

Pull gives you pacing, not unlimited backpressure

Consumers fetch records in batches and control their processing pace. Fetch-size settings and max.poll.records help control work, but the latter limits records returned by a poll, not the underlying fetch cache. The consumer configuration reference distinguishes these controls.

A slow consumer accumulates lag: its progress falls behind the log's end. Pull consumption doesn't stop retention cleanup, so a consumer can fall behind far enough that the records it needs are gone.

Membership changes and partition changes can trigger rebalancing: reassignment of partitions between consumers. Partition handoffs can affect processing and latency. Kafka's newer consumer rebalance protocol uses incremental assignment without a group-wide synchronization barrier; don't assume every rebalance stops every member.

Offsets, Restart Recovery, and Processing Guarantees

A consumer's current position advances as records are returned. Its committed position is the checkpoint used for recovery. Consumers can commit automatically or explicitly; Kafka's group coordinator normally stores those commits in the internal __consumer_offsets topic, as described in consumer offset tracking.

A committed offset identifies the next position to consume, not the last record successfully processed. Restarting from a checkpoint works only if that position is still available; offset-reset policy determines what happens when there is no usable checkpoint.

# Pseudocode: manual commits and synchronous batch processing
repeat:
    batch = fetch_records()
    process_every_record_or_stop(batch)
    commit_next_offsets(batch)

# Don't advance the checkpoint past a failed record.
# Handle commit failures and partition revocation.

For Kafka-to-Kafka processing, transactions can atomically commit output records and consumed offsets; downstream consumers use isolation.level=read_committed to exclude aborted transactional output. That doesn't automatically make a database write or payment request atomic with the Kafka transaction. External systems require cooperation, such as transactional offset storage or idempotent effects.

The delivery-semantics design section explains these boundaries. When implementing the consumer side in Spring, continue with the Spring Boot Kafka consumer guide.

Kafka Retention and Log Compaction

Reading a record doesn't remove it. With deletion-based retention, the broker defaults provide 168 hours of time retention and no byte-based limit: log.retention.bytes=-1. Topic overrides and broker configuration determine the effective policy; these aren't universal settings for every deployed cluster.

Size-based retention.bytes applies per partition, not to the topic as a whole. Segment-based cleanup also means retention isn't an exact per-record deletion timer. The topic configuration reference documents the effective controls.

Compaction preserves the latest known record per key over time; it doesn't immediately leave exactly one record per key. A keyed record with a null value is a tombstone, marking deletion. Tombstones later become eligible for removal under delete.retention.ms, so state-rebuilding consumers must account for cleanup timing.

Core Kafka Versus Connect, Streams, and Databases

Kafka Connect runs connectors that move data between Kafka and external systems. Kafka Streams is an application library for processing streams, including stateful operations backed by local state stores and recovery changelogs. Neither replaces the broker's partition-log architecture.

Kafka can support event sourcing and durable changelogs, but the broker isn't a general-purpose query database. Indexed application queries belong in a database or suitable state store; a retained event log doesn't supply relational constraints or arbitrary query semantics.

Common Mistakes

• Scaling consumers without checking partitions: extra group members don't fix an assignment-parallelism limit or a hot partition.
• Changing partition counts casually: hash-based key mappings can change, while existing records remain in their old partitions. Plan for the ordering implications of partition expansion.
• Committing after dispatch rather than completion: handing work to another thread isn't proof that its side effect succeeded.
• Testing only the happy path: exercise crashes between processing and commit, failed sends, and partition reassignment. See the Spring Kafka testing examples for application-level testing.

The architectural trade-off is straightforward: partitions supply scalable logs, replication supplies redundant storage, and offsets supply independent progress. Your application still owns its data contract, processing behavior, and recovery policy.

FAQ

What are the main components of Kafka architecture?

Kafka's main components are producers, topics and partitions, brokers, consumer groups, and KRaft controllers. Brokers store replicated event logs, while controllers manage cluster metadata and partition leadership.

Can one Kafka consumer read from multiple partitions?

Yes. In a standard consumer group, one consumer can be assigned several partitions, but each partition has at most one assigned consumer in that group. Separate groups can read the same partitions independently.

Does Kafka 4.x use ZooKeeper?

No. Kafka 4.0 removed ZooKeeper mode, and Kafka 4.x uses KRaft for metadata management. ZooKeeper-based clusters must migrate to KRaft before upgrading to Kafka 4.x.

What is the difference between Kafka retention and log compaction?

Deletion-based retention removes old log segments according to time or size limits. Compaction asynchronously removes superseded records for each key, while tombstones represent deletions. A topic can enable both policies.

Does Kafka guarantee exactly-once processing?

Kafka supports exactly-once Kafka-visible results through transactions that combine output records and consumed offsets, with appropriate consumer isolation. This doesn't automatically extend to external database writes or API calls; those effects require additional coordination or deduplication.

Further Reading

• Apache Kafka introduction — events, topics, partitions, and independent consumption.
• KRaft operations — controller roles and metadata quorums.
• Kafka design — storage, replication, compaction, and delivery semantics.
• Producer configuration — partition selection, batching, and acknowledgments.
• KafkaConsumer reference — group assignment, positions, and offset commits.
• Broker configuration — replication requirements and retention defaults.
• Topic configuration — cleanup policies and topic-level overrides.
• Kafka Streams architecture — processing tasks, state stores, and recovery.