Apache Kafka Explained: Topics, Partitions, Consumer Groups and When to Use It

Apache Kafka sits at the center of many modern data and event-driven architectures. It moves huge volumes of events between systems with durability and ordering guarantees, and lets many consumers read the same data independently. But Kafka is also operationally demanding and often adopted where a simpler queue would do.

This guide explains how Kafka works, what it's good for, and the pitfalls to avoid.

#Key takeaways

• Kafka is a distributed, durable, append-only log. Producers append events; consumers read them at their own pace.
• Topics are split into partitions; ordering is guaranteed within a partition, not across a topic.
• Consumer groups divide partitions among consumers for parallelism; different groups read the same data independently.
• Events are retained for a configured time or size, so consumers can replay history.
• Use Kafka for high-throughput event streaming, fan-out and replay. For simple task queues, lighter tools are often better (message queues explained).

#The core idea: a log

Unlike traditional queues where a message disappears once consumed, Kafka stores events in an ordered, append-only log:

textPartition 0:  [0] [1] [2] [3] [4] [5] [6] [7] ...  ← producers append here
                         ▲               ▲
              consumer group A      consumer group B
              (offset 3)            (offset 6)

Each event has an offset, its position in the partition. Consumers track which offset they've processed. Reading doesn't delete anything; events expire according to retention policy.

That design unlocks three powerful properties:

1. Multiple independent consumers: analytics, search indexing and notifications can all read the same events.
2. Replay: a new consumer, or a fixed buggy one, can reprocess from an earlier offset.
3. Decoupling: producers don't know or care who consumes.

#Topics, partitions and keys

A topic is a named stream of events (orders, page_views). Each topic is split into partitions, which are the unit of parallelism and ordering.

When producing, you can set a key. Kafka hashes the key to choose the partition, so all events with the same key go to the same partition, in order:

pythonproducer.send("orders", key=b"customer_42", value=order_event_bytes)

• Key by customer_id → all of a customer's events stay in order.
• No key → events are spread across partitions for throughput, with no ordering across them.

Choosing the partition count and key is a design decision: too few partitions limits parallelism; a skewed key (one huge customer) creates hot partitions.

#Brokers and replication

A Kafka cluster consists of multiple brokers (servers). Each partition has one leader and several follower replicas on other brokers. Producers write to the leader; followers replicate. If a broker fails, a follower becomes the leader.

Key settings:

• Replication factor: commonly 3 for production topics.
• min.insync.replicas: how many replicas must acknowledge a write.
• Producer acks=all: the write succeeds only when the in-sync replicas have it. Use it for durability.

Modern Kafka manages cluster metadata with its built-in KRaft consensus mode, which removed the older dependency on ZooKeeper.

#Consumer groups

Consumers that share a group ID split a topic's partitions among themselves:

textTopic "orders" (6 partitions)
Consumer group "billing":   C1 → P0,P1   C2 → P2,P3   C3 → P4,P5
Consumer group "analytics": C1 → P0..P5  (independent offsets)

• Each partition is consumed by exactly one consumer in a group at a time, which preserves per-partition ordering.
• Adding consumers (up to the partition count) increases parallelism.
• When consumers join or leave, Kafka rebalances partition assignments.
• Consumers commit offsets to record progress.

#Delivery guarantees

In practice: design consumers to be idempotent (processing the same event twice has no extra effect) and use at-least-once delivery. Kafka's exactly-once semantics apply to read-process-write flows within Kafka; side effects in external systems still need idempotency keys or deduplication.

#Common use cases

• Event-driven microservices: services publish domain events (OrderPlaced, PaymentFailed) that others react to (event-driven architecture).
• Data pipelines: streaming app events and database changes into warehouses and lakes.
• Change data capture: database change events flowing through Kafka to downstream systems (CDC explained).
• Real-time analytics and monitoring: clickstreams, metrics, fraud signals.
• Log aggregation: centralizing logs from many services.
• Stream processing: joining, aggregating and enriching streams in real time (batch vs stream processing).

#Schemas and compatibility

Events are contracts between teams. Without schemas, a producer's field rename silently breaks consumers. Use a serialization format with schemas (Avro, Protobuf or JSON Schema) and a schema registry that enforces compatibility rules, such as allowing new optional fields but not removing required ones.

#Operational realities

Kafka is powerful and not trivial to run:

• Capacity planning: partitions, disk, network and retention all interact.
• Consumer lag monitoring: how far behind consumers are is the key health metric.
• Rebalancing storms can pause consumption; tune session timeouts and use cooperative rebalancing.
• Partition count is hard to reduce later, and increasing it changes key-to-partition mapping.
• Upgrades, security (TLS, ACLs) and multi-region replication need expertise.

Managed Kafka services and Kafka-compatible platforms remove much of the operational burden. For small teams, they're usually worth it.

#When not to use Kafka

• Simple background jobs (send email, resize image): a job queue or a managed queue is simpler.
• Low volume with no need for replay or fan-out.
• Request/response interactions: use HTTP or gRPC.
• Complex routing per message: traditional message brokers handle routing patterns more naturally.

#Kafka vs traditional message queues

#FAQ

#Is Kafka a database?

Not really. It durably stores events and can retain them for long periods, but it's optimized for sequential reads and writes, not ad-hoc queries or random updates. It typically feeds databases rather than replacing them.

#How many partitions should a topic have?

Enough to support your target consumer parallelism and throughput, with some headroom. Too many partitions add overhead. Estimate from your expected throughput per partition and plan for growth, since reducing partitions later isn't supported.

#Does Kafka guarantee message order?

Only within a partition. If ordering matters for an entity (like a customer's orders), use that entity's ID as the message key so its events land in the same partition.

#What is consumer lag?

The difference between the latest offset in a partition and the consumer group's committed offset: how many events are waiting to be processed. Rising lag means consumers can't keep up, and it's the first metric to alert on.